[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:25:52 dist-test-slave-vq9v krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:25:52 dist-test-slave-vq9v krb5kdc[399](info): set up 2 sockets
Sep 29 16:25:52 dist-test-slave-vq9v krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 16:25:53.494356 392 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:25:53 dist-test-slave-vq9v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:25:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:25:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:25:53.622175 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36099
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:25:53.734639 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:25:53.734997 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:25:53.735066 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:25:53.739380 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:25:53.739496 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:25:53.739537 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:25:53.739598 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:25:53.739636 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:25:53.745687 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:25:53.746938 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:25:53.748109 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:25:53.758581 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:53.758632 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:53.758668 412 server_base.cc:1034] running on GCE node
W20260929 16:25:53.758605 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:53.759547 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:25:53.760921 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:25:53.762137 412 hybrid_clock.cc:648] HybridClock initialized: now 1790699153762064 us; error 91 us; skew 500 ppm
Sep 29 16:25:53 dist-test-slave-vq9v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:25:53.765440 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:25:53.766533 412 webserver.cc:460] Webserver started at http://127.0.98.62:35837/ using document root <none> and password file <none>
I20260929 16:25:53.767196 412 fs_manager.cc:359] Metadata directory not provided
I20260929 16:25:53.767305 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:25:53.767577 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:25:53.769915 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "86e9e5d09c27403fa66ccde7d26b31e6"
format_stamp: "Formatted at 2026-09-29 16:25:53 on dist-test-slave-vq9v"
server_key: "cf2890148c958fc85089ffd6ba6f0317"
server_key_iv: "4445860d419f98bfb1582242f144834d"
server_key_version: "encryptionkey@0"
I20260929 16:25:53.770561 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "86e9e5d09c27403fa66ccde7d26b31e6"
format_stamp: "Formatted at 2026-09-29 16:25:53 on dist-test-slave-vq9v"
server_key: "cf2890148c958fc85089ffd6ba6f0317"
server_key_iv: "4445860d419f98bfb1582242f144834d"
server_key_version: "encryptionkey@0"
I20260929 16:25:53.774945 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:25:53.777781 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:53.779383 412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:25:53.779558 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "86e9e5d09c27403fa66ccde7d26b31e6"
format_stamp: "Formatted at 2026-09-29 16:25:53 on dist-test-slave-vq9v"
server_key: "cf2890148c958fc85089ffd6ba6f0317"
server_key_iv: "4445860d419f98bfb1582242f144834d"
server_key_version: "encryptionkey@0"
I20260929 16:25:53.779703 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:25:53.799315 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:25:53.803057 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:25:53.803311 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:25:53.810712 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36099
I20260929 16:25:53.810779 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36099 every 8 connection(s)
I20260929 16:25:53.811779 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:25:53.815171 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:25:53.819733 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 412
I20260929 16:25:53.819983 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:25:53.821357 392 external_mini_cluster.cc:1428] Setting key e502ba3ea6bfa5e27aa3d5fc9045293d
I20260929 16:25:53.821811 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6: Bootstrap starting.
I20260929 16:25:53.825140 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6: Neither blocks nor log segments found. Creating new log.
I20260929 16:25:53.826237 480 log.cc:824] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:25:53.828307 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6: No bootstrap required, opened a new log
I20260929 16:25:53.832836 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e9e5d09c27403fa66ccde7d26b31e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36099 } }
I20260929 16:25:53.833150 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:25:53.833292 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86e9e5d09c27403fa66ccde7d26b31e6, State: Initialized, Role: FOLLOWER
I20260929 16:25:53.833806 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e9e5d09c27403fa66ccde7d26b31e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36099 } }
I20260929 16:25:53.833981 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:25:53.834080 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:25:53.834208 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:25:53.835605 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e9e5d09c27403fa66ccde7d26b31e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36099 } }
I20260929 16:25:53.836054 480 leader_election.cc:302] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86e9e5d09c27403fa66ccde7d26b31e6; no voters:
I20260929 16:25:53.836486 480 leader_election.cc:288] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:25:53.836583 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:25:53.836915 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6 [term 1 LEADER]: Becoming Leader. State: Replica: 86e9e5d09c27403fa66ccde7d26b31e6, State: Running, Role: LEADER
I20260929 16:25:53.837325 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e9e5d09c27403fa66ccde7d26b31e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36099 } }
I20260929 16:25:53.838039 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:25:53.838716 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86e9e5d09c27403fa66ccde7d26b31e6. Latest consensus state: current_term: 1 leader_uuid: "86e9e5d09c27403fa66ccde7d26b31e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e9e5d09c27403fa66ccde7d26b31e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36099 } } }
I20260929 16:25:53.838867 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:25:53.839567 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86e9e5d09c27403fa66ccde7d26b31e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e9e5d09c27403fa66ccde7d26b31e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36099 } } }
I20260929 16:25:53.839666 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:25:53.839977 491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 29 16:25:53 dist-test-slave-vq9v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:25:53.844666 491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:25:53.849884 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:53.824514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.0.98.62:36099)
0929 16:25:53.824985 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:53.825001 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:53.826450 (+ 1449us) server_negotiation.cc:407] Connection header received
0929 16:25:53.829757 (+ 3307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:53.829773 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:53.830091 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:53.830433 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:53.832681 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:53.833944 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:53.836049 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:53.836771 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:53.836981 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:53.842920 (+ 5939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:53.842927 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:53.842945 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:53.842984 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:53.845250 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:53.846103 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:53.846110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:53.846116 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:53.846308 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:53.846830 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:53.846833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:53.846835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:53.847391 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:53.847668 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:53.848385 (+ 717us) server_negotiation.cc:299] Negotiation successful
0929 16:25:53.848706 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":153,"threads_started":1}
I20260929 16:25:53.853161 491 catalog_manager.cc:1347] Generated new cluster ID: 7fd80fc198ff4017842a617bb4228e86
I20260929 16:25:53.853292 491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:25:53.885141 491 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:25:53.886512 491 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:25:53.893666 491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86e9e5d09c27403fa66ccde7d26b31e6: Generated new TSK 0
I20260929 16:25:53.894457 491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 16:25:53.936949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36099
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:25:54.066618 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:25:54.066949 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:25:54.066998 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:25:54.071735 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:25:54.071847 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:25:54.071915 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:25:54.077775 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36099
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:25:54.078895 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:25:54.079933 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:25:54.090173 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:54.090227 506 server_base.cc:1034] running on GCE node
W20260929 16:25:54.090179 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:54.090167 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:54.090906 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:25:54.091660 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:25:54.092869 506 hybrid_clock.cc:648] HybridClock initialized: now 1790699154092836 us; error 55 us; skew 500 ppm
Sep 29 16:25:54 dist-test-slave-vq9v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:25:54.096140 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:25:54.097110 506 webserver.cc:460] Webserver started at http://127.0.98.1:44905/ using document root <none> and password file <none>
I20260929 16:25:54.097970 506 fs_manager.cc:359] Metadata directory not provided
I20260929 16:25:54.098074 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:25:54.098316 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:25:54.100498 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "748c6ab723f1400489ade46150a3eac8"
format_stamp: "Formatted at 2026-09-29 16:25:54 on dist-test-slave-vq9v"
server_key: "79aed60d22cf8576bbb5bf13b409896e"
server_key_iv: "77f0cb54a9a3e465c3236ad92258656c"
server_key_version: "encryptionkey@0"
I20260929 16:25:54.101130 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "748c6ab723f1400489ade46150a3eac8"
format_stamp: "Formatted at 2026-09-29 16:25:54 on dist-test-slave-vq9v"
server_key: "79aed60d22cf8576bbb5bf13b409896e"
server_key_iv: "77f0cb54a9a3e465c3236ad92258656c"
server_key_version: "encryptionkey@0"
I20260929 16:25:54.105432 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:25:54.108327 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:54.109966 506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:25:54.110129 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "748c6ab723f1400489ade46150a3eac8"
format_stamp: "Formatted at 2026-09-29 16:25:54 on dist-test-slave-vq9v"
server_key: "79aed60d22cf8576bbb5bf13b409896e"
server_key_iv: "77f0cb54a9a3e465c3236ad92258656c"
server_key_version: "encryptionkey@0"
I20260929 16:25:54.110262 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:25:54.132185 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:25:54.135900 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:25:54.136117 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:25:54.136741 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:25:54.137671 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:25:54.137743 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:54.137807 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:25:54.137861 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:54.148170 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42823
I20260929 16:25:54.148190 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42823 every 8 connection(s)
I20260929 16:25:54.149125 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:25:54.155256 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 506
I20260929 16:25:54.155354 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:25:54.155658 392 external_mini_cluster.cc:1428] Setting key 5384fc2708e5af5c919f95399e23a344
Sep 29 16:25:54 dist-test-slave-vq9v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 16:25:54.165997 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:54.151633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47529 (local address 127.0.98.62:36099)
0929 16:25:54.151805 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:54.151810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:54.152721 (+ 911us) server_negotiation.cc:407] Connection header received
0929 16:25:54.154112 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:54.154115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:54.154161 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:54.154237 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:54.155601 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:54.156272 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:54.157966 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:54.158195 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:54.158310 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:54.161783 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:54.161787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:54.161789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:54.161817 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:54.163534 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:54.164195 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:54.164198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:54.164199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:54.164285 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:54.164673 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:54.164676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:54.164678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:54.164880 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:54.164994 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:54.165668 (+ 674us) server_negotiation.cc:299] Negotiation successful
0929 16:25:54.165824 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:25:54.166989 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:54.152084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36099 (local address 127.0.98.1:47529)
0929 16:25:54.152557 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:54.152603 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:54.153911 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:54.154451 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:54.154458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:54.154848 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:54.155450 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:54.155466 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:54.156411 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:54.156415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:54.157809 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:54.157836 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:54.158385 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:54.158391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:54.158643 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:54.159532 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:54.159552 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:54.161538 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:54.163703 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:54.163712 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:54.163733 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:54.164055 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:54.164388 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:54.164391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:54.164393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:54.164560 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:54.164994 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:54.165004 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:54.165365 (+ 361us) client_negotiation.cc:769] Sending connection context
0929 16:25:54.165611 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 16:25:54.165993 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 16:25:54.168421 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36099
I20260929 16:25:54.168726 635 heartbeater.cc:464] Registering TS with master...
I20260929 16:25:54.169456 635 heartbeater.cc:511] Master 127.0.98.62:36099 requested a full tablet report, sending...
I20260929 16:25:54.171456 444 ts_manager.cc:194] Registered new tserver with Master: 748c6ab723f1400489ade46150a3eac8 (127.0.98.1:42823)
I20260929 16:25:54.172710 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 16:25:54.189160 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36099
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:25:54.311772 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:25:54.312134 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:25:54.312219 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:25:54.316737 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:25:54.316844 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:25:54.316948 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:25:54.322715 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36099
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:25:54.323912 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:25:54.325063 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:25:54.335476 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:54.335467 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:54.335465 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:54.335837 640 server_base.cc:1034] running on GCE node
I20260929 16:25:54.336342 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:25:54.336989 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:25:54.338176 640 hybrid_clock.cc:648] HybridClock initialized: now 1790699154338165 us; error 66 us; skew 500 ppm
Sep 29 16:25:54 dist-test-slave-vq9v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:25:54.341512 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:25:54.342509 640 webserver.cc:460] Webserver started at http://127.0.98.2:39447/ using document root <none> and password file <none>
I20260929 16:25:54.343106 640 fs_manager.cc:359] Metadata directory not provided
I20260929 16:25:54.343225 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:25:54.343488 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:25:54.345983 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "2156da6281ff49038609e814f12b7a73"
format_stamp: "Formatted at 2026-09-29 16:25:54 on dist-test-slave-vq9v"
server_key: "78846db7ecaf36035980e71c0254a1f6"
server_key_iv: "0e53add7aa113a1ee64410415805f80c"
server_key_version: "encryptionkey@0"
I20260929 16:25:54.346611 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2156da6281ff49038609e814f12b7a73"
format_stamp: "Formatted at 2026-09-29 16:25:54 on dist-test-slave-vq9v"
server_key: "78846db7ecaf36035980e71c0254a1f6"
server_key_iv: "0e53add7aa113a1ee64410415805f80c"
server_key_version: "encryptionkey@0"
I20260929 16:25:54.351063 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:25:54.354254 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:54.355978 640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:25:54.356135 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "2156da6281ff49038609e814f12b7a73"
format_stamp: "Formatted at 2026-09-29 16:25:54 on dist-test-slave-vq9v"
server_key: "78846db7ecaf36035980e71c0254a1f6"
server_key_iv: "0e53add7aa113a1ee64410415805f80c"
server_key_version: "encryptionkey@0"
I20260929 16:25:54.356304 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:25:54.392356 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:25:54.396078 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:25:54.396308 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:25:54.396935 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:25:54.397836 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:25:54.397912 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:54.397984 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:25:54.398028 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:54.409487 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45259
I20260929 16:25:54.409514 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45259 every 8 connection(s)
I20260929 16:25:54.410586 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:25:54.416822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 640
I20260929 16:25:54.416983 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:25:54.417337 392 external_mini_cluster.cc:1428] Setting key 52ae479dc6851c2973aacd36287e8bdc
Sep 29 16:25:54 dist-test-slave-vq9v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:25:54.426659 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:54.412577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38345 (local address 127.0.98.62:36099)
0929 16:25:54.412801 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:54.412805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:54.413693 (+ 888us) server_negotiation.cc:407] Connection header received
0929 16:25:54.415268 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:54.415271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:54.415335 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:54.415413 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:54.416684 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:54.417417 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:54.418797 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:54.419122 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:54.419293 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:54.421939 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:54.421946 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:54.421952 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:54.421991 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:54.423968 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:54.424552 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:54.424558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:54.424562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:54.424666 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:54.425261 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:54.425267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:54.425270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:54.425525 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:54.425642 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:54.426306 (+ 664us) server_negotiation.cc:299] Negotiation successful
0929 16:25:54.426446 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 16:25:54.427816 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:54.412984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36099 (local address 127.0.98.2:38345)
0929 16:25:54.413534 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:54.413580 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:54.415009 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:54.415607 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:54.415615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:54.416011 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:54.416514 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:54.416528 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:54.417556 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:54.417566 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:54.418637 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:54.418647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:54.419322 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:54.419326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:54.419479 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:54.420146 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:54.420164 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:54.421754 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:54.424124 (+ 2370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:54.424129 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:54.424145 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:54.424438 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:54.424948 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:54.424950 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:54.424952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:54.425111 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:54.425687 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:54.425694 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:54.426008 (+ 314us) client_negotiation.cc:769] Sending connection context
0929 16:25:54.426231 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 16:25:54.426570 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":165,"threads_started":1}
I20260929 16:25:54.429406 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36099
I20260929 16:25:54.429774 769 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 16:25:54.430514 769 heartbeater.cc:511] Master 127.0.98.62:36099 requested a full tablet report, sending...
I20260929 16:25:54.432200 444 ts_manager.cc:194] Registered new tserver with Master: 2156da6281ff49038609e814f12b7a73 (127.0.98.2:45259)
I20260929 16:25:54.432868 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 16:25:54.450124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36099
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:25:54.566766 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:25:54.567106 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:25:54.567183 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:25:54.571488 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:25:54.571612 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:25:54.571722 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:25:54.577543 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36099
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:25:54.578733 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:25:54.579880 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:25:54.591424 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:54.591424 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:54.591563 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:54.592070 774 server_base.cc:1034] running on GCE node
I20260929 16:25:54.592609 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:25:54.593436 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:25:54.594671 774 hybrid_clock.cc:648] HybridClock initialized: now 1790699154594634 us; error 59 us; skew 500 ppm
Sep 29 16:25:54 dist-test-slave-vq9v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:25:54.597718 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:25:54.598661 774 webserver.cc:460] Webserver started at http://127.0.98.3:37945/ using document root <none> and password file <none>
I20260929 16:25:54.599207 774 fs_manager.cc:359] Metadata directory not provided
I20260929 16:25:54.599269 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:25:54.599436 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:25:54.601528 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "edf1dac102704112a6892c5ffb9cb8d3"
format_stamp: "Formatted at 2026-09-29 16:25:54 on dist-test-slave-vq9v"
server_key: "614ebfed2496edff72bd42768c7346bc"
server_key_iv: "8f52908f9152cf3dae5135c056ab1a15"
server_key_version: "encryptionkey@0"
I20260929 16:25:54.602041 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "edf1dac102704112a6892c5ffb9cb8d3"
format_stamp: "Formatted at 2026-09-29 16:25:54 on dist-test-slave-vq9v"
server_key: "614ebfed2496edff72bd42768c7346bc"
server_key_iv: "8f52908f9152cf3dae5135c056ab1a15"
server_key_version: "encryptionkey@0"
I20260929 16:25:54.605852 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 16:25:54.608579 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:54.610018 774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:25:54.610162 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "edf1dac102704112a6892c5ffb9cb8d3"
format_stamp: "Formatted at 2026-09-29 16:25:54 on dist-test-slave-vq9v"
server_key: "614ebfed2496edff72bd42768c7346bc"
server_key_iv: "8f52908f9152cf3dae5135c056ab1a15"
server_key_version: "encryptionkey@0"
I20260929 16:25:54.610289 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:25:54.628058 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:25:54.631762 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:25:54.631989 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:25:54.632624 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:25:54.633520 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:25:54.633594 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:54.633659 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:25:54.633714 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:54.644606 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46249
I20260929 16:25:54.644645 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46249 every 8 connection(s)
I20260929 16:25:54.645707 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:25:54.646903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 774
I20260929 16:25:54.647001 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:25:54.647310 392 external_mini_cluster.cc:1428] Setting key 4b6495c70ebcc7d55897685ca6596c96
Sep 29 16:25:54 dist-test-slave-vq9v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:25:54.661264 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:54.647626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54265 (local address 127.0.98.62:36099)
0929 16:25:54.647783 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:54.647788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:54.648999 (+ 1211us) server_negotiation.cc:407] Connection header received
0929 16:25:54.650771 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:54.650774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:54.650829 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:54.650922 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:54.652266 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:54.652870 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:54.654106 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:54.654339 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:54.654437 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:54.657043 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:54.657047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:54.657049 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:54.657078 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:54.658566 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:54.659105 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:54.659108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:54.659109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:54.659185 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:54.659601 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:54.659604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:54.659606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:54.659853 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:54.659948 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:54.660802 (+ 854us) server_negotiation.cc:299] Negotiation successful
0929 16:25:54.661009 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:25:54.662322 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:54.648166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36099 (local address 127.0.98.3:54265)
0929 16:25:54.648720 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:54.648777 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:54.650437 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:54.651091 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:54.651099 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:54.651489 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:54.652035 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:54.652050 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:54.652999 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:54.653002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:54.653987 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:54.653996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:54.654481 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:54.654483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:54.654627 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:54.655245 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:54.655262 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:54.656844 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:54.658693 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:54.658698 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:54.658711 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:54.659007 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:54.659295 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:54.659298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:54.659299 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:54.659478 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:54.659975 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:54.659980 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:54.660387 (+ 407us) client_negotiation.cc:769] Sending connection context
0929 16:25:54.660712 (+ 325us) client_negotiation.cc:240] Negotiation successful
0929 16:25:54.661117 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":144,"threads_started":1}
I20260929 16:25:54.663528 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36099
I20260929 16:25:54.663849 903 heartbeater.cc:464] Registering TS with master...
I20260929 16:25:54.664430 903 heartbeater.cc:511] Master 127.0.98.62:36099 requested a full tablet report, sending...
I20260929 16:25:54.665634 444 ts_manager.cc:194] Registered new tserver with Master: edf1dac102704112a6892c5ffb9cb8d3 (127.0.98.3:46249)
I20260929 16:25:54.666252 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54265
I20260929 16:25:54.675140 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:25:54 dist-test-slave-vq9v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699154, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:25:54 dist-test-slave-vq9v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699154, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 16:25:54.704690 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:54.693867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33324 (local address 127.0.98.1:42823)
0929 16:25:54.694134 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:54.694144 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:54.694180 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:25:54.694271 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:54.694276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:54.694470 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:54.694602 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:54.695375 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:54.696217 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:54.697284 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:54.697767 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:54.697971 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:54.700597 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:54.700609 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:54.700624 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:54.700666 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:54.702580 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:54.703204 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:54.703210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:54.703211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:54.703338 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:54.703657 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:54.703659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:54.703660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:54.703935 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:54.704109 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:54.704388 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 16:25:54.704533 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":103,"threads_started":1}
W20260929 16:25:54.705904 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33324
Sep 29 16:25:54 dist-test-slave-vq9v krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:25:54 dist-test-slave-vq9v krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 16:25:54.734788 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:54.724381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33336 (local address 127.0.98.1:42823)
0929 16:25:54.724551 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:54.724555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:54.724701 (+ 146us) server_negotiation.cc:407] Connection header received
0929 16:25:54.724832 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:54.724834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:54.724880 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:54.724968 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:54.726057 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:54.726619 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:54.727682 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:54.727919 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:54.728022 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:54.731085 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:54.731089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:54.731091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:54.731125 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:54.732801 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:54.733388 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:54.733391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:54.733392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:54.733486 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:54.733902 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:54.733905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:54.733906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:54.734115 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:54.734248 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:54.734534 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 16:25:54.734652 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 16:25:54.736097 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 506
I20260929 16:25:54.742271 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 640
I20260929 16:25:54.751003 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 774
I20260929 16:25:54.757923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 412
2026-09-29T16:25:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2355 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:25:54 dist-test-slave-vq9v krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:25:54 dist-test-slave-vq9v krb5kdc[917](info): set up 2 sockets
Sep 29 16:25:54 dist-test-slave-vq9v krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260929 16:25:55.805092 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:25:55 dist-test-slave-vq9v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:25:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:25:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:25:55.942622 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37559
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:25:56.089645 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:25:56.090024 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:25:56.090113 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:25:56.094573 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:25:56.094679 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:25:56.094718 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:25:56.094772 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:25:56.094822 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:25:56.101143 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:25:56.102444 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:25:56.103559 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:25:56.113668 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:56.113686 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:56.113668 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:56.114097 930 server_base.cc:1034] running on GCE node
I20260929 16:25:56.114794 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:25:56.116073 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:25:56.117324 930 hybrid_clock.cc:648] HybridClock initialized: now 1790699156117272 us; error 66 us; skew 500 ppm
Sep 29 16:25:56 dist-test-slave-vq9v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:25:56.120857 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:25:56.122020 930 webserver.cc:460] Webserver started at http://127.0.98.62:34335/ using document root <none> and password file <none>
I20260929 16:25:56.122630 930 fs_manager.cc:359] Metadata directory not provided
I20260929 16:25:56.122702 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:25:56.122901 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:25:56.125497 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "495df05dfc804b08a58f67022260367c"
format_stamp: "Formatted at 2026-09-29 16:25:56 on dist-test-slave-vq9v"
server_key: "3b4bb3b9e5bad6da846e7eee96cd6ca8"
server_key_iv: "8306579da6a290443a8a755d56d87ab5"
server_key_version: "encryptionkey@0"
I20260929 16:25:56.126125 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "495df05dfc804b08a58f67022260367c"
format_stamp: "Formatted at 2026-09-29 16:25:56 on dist-test-slave-vq9v"
server_key: "3b4bb3b9e5bad6da846e7eee96cd6ca8"
server_key_iv: "8306579da6a290443a8a755d56d87ab5"
server_key_version: "encryptionkey@0"
I20260929 16:25:56.130573 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:25:56.133550 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:56.135255 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:25:56.135398 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "495df05dfc804b08a58f67022260367c"
format_stamp: "Formatted at 2026-09-29 16:25:56 on dist-test-slave-vq9v"
server_key: "3b4bb3b9e5bad6da846e7eee96cd6ca8"
server_key_iv: "8306579da6a290443a8a755d56d87ab5"
server_key_version: "encryptionkey@0"
I20260929 16:25:56.135511 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:25:56.153110 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:25:56.157001 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:25:56.157171 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:25:56.165098 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37559
I20260929 16:25:56.165124 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37559 every 8 connection(s)
I20260929 16:25:56.166242 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:25:56.169384 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:25:56.175324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 930
I20260929 16:25:56.175493 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:25:56.175731 392 external_mini_cluster.cc:1428] Setting key 11619993cf90fcf0ae4454c4bce74682
I20260929 16:25:56.176111 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c: Bootstrap starting.
I20260929 16:25:56.178998 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c: Neither blocks nor log segments found. Creating new log.
I20260929 16:25:56.179920 998 log.cc:824] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:25:56.182742 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c: No bootstrap required, opened a new log
Sep 29 16:25:56 dist-test-slave-vq9v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:25:56.188220 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495df05dfc804b08a58f67022260367c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37559 } }
I20260929 16:25:56.188572 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:25:56.188633 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495df05dfc804b08a58f67022260367c, State: Initialized, Role: FOLLOWER
I20260929 16:25:56.189278 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495df05dfc804b08a58f67022260367c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37559 } }
I20260929 16:25:56.189479 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:25:56.189543 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:25:56.189636 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:25:56.190944 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495df05dfc804b08a58f67022260367c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37559 } }
I20260929 16:25:56.191521 998 leader_election.cc:302] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 495df05dfc804b08a58f67022260367c; no voters:
I20260929 16:25:56.191991 998 leader_election.cc:288] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:25:56.192135 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:25:56.192404 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c [term 1 LEADER]: Becoming Leader. State: Replica: 495df05dfc804b08a58f67022260367c, State: Running, Role: LEADER
I20260929 16:25:56.192862 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495df05dfc804b08a58f67022260367c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37559 } }
I20260929 16:25:56.193599 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:25:56.194427 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:56.177573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47008 (local address 127.0.98.62:37559)
0929 16:25:56.178095 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:56.178112 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:56.178151 (+ 39us) server_negotiation.cc:407] Connection header received
0929 16:25:56.179069 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:56.179094 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:56.179610 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:56.179955 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:56.181026 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:56.182224 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:56.183715 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:56.184240 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:56.184401 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:56.187882 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:56.187889 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:56.187919 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:56.187959 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:56.190194 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:56.190783 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:56.190794 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:56.190804 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:56.190982 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:56.191372 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:56.191375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:56.191377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:56.192236 (+ 859us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:56.192460 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:56.192849 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 16:25:56.193240 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":190,"threads_started":1}
I20260929 16:25:56.194957 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 495df05dfc804b08a58f67022260367c. Latest consensus state: current_term: 1 leader_uuid: "495df05dfc804b08a58f67022260367c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495df05dfc804b08a58f67022260367c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37559 } } }
I20260929 16:25:56.195116 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c [sys.catalog]: This master's current role is: LEADER
I20260929 16:25:56.195500 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "495df05dfc804b08a58f67022260367c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495df05dfc804b08a58f67022260367c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37559 } } }
I20260929 16:25:56.195590 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c [sys.catalog]: This master's current role is: LEADER
W20260929 16:25:56.199777 1018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:25:56.199891 1018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:25:56.200011 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:25:56.200848 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:25:56.212261 1012 catalog_manager.cc:1347] Generated new cluster ID: 431bdff9656a421599010712aa96bf62
I20260929 16:25:56.212363 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:25:56.248590 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:25:56.249809 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:25:56.266601 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 495df05dfc804b08a58f67022260367c: Generated new TSK 0
I20260929 16:25:56.267983 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260929 16:25:56.716054 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.433s user 0.003s sys 0.001s
I20260929 16:25:56.716508 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37559
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:25:56.830765 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:25:56.831135 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:25:56.831218 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:25:56.835520 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:25:56.835635 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:25:56.835736 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:25:56.841741 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37559
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:25:56.842921 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:25:56.844056 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:25:56.856215 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:56.856163 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:56.856132 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:56.856429 1024 server_base.cc:1034] running on GCE node
I20260929 16:25:56.856998 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:25:56.857860 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:25:56.859208 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790699156859088 us; error 40 us; skew 500 ppm
Sep 29 16:25:56 dist-test-slave-vq9v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:25:56.870005 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:25:56.871501 1024 webserver.cc:460] Webserver started at http://127.0.98.1:41605/ using document root <none> and password file <none>
I20260929 16:25:56.872682 1024 fs_manager.cc:359] Metadata directory not provided
I20260929 16:25:56.872807 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:25:56.873091 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:25:56.876951 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "d9ff4842c7494a76bb16c9a7aed6ae6d"
format_stamp: "Formatted at 2026-09-29 16:25:56 on dist-test-slave-vq9v"
server_key: "ac8be5e40a5bf8f276abf47f70251bed"
server_key_iv: "80799f380163b642e4b6bcb655db974f"
server_key_version: "encryptionkey@0"
I20260929 16:25:56.878165 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d9ff4842c7494a76bb16c9a7aed6ae6d"
format_stamp: "Formatted at 2026-09-29 16:25:56 on dist-test-slave-vq9v"
server_key: "ac8be5e40a5bf8f276abf47f70251bed"
server_key_iv: "80799f380163b642e4b6bcb655db974f"
server_key_version: "encryptionkey@0"
I20260929 16:25:56.884653 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 16:25:56.888263 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:56.890276 1024 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:25:56.890444 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "d9ff4842c7494a76bb16c9a7aed6ae6d"
format_stamp: "Formatted at 2026-09-29 16:25:56 on dist-test-slave-vq9v"
server_key: "ac8be5e40a5bf8f276abf47f70251bed"
server_key_iv: "80799f380163b642e4b6bcb655db974f"
server_key_version: "encryptionkey@0"
I20260929 16:25:56.890614 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:25:56.919478 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:25:56.923594 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:25:56.923818 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:25:56.924486 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:25:56.925441 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:25:56.925491 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:56.925557 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:25:56.925587 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:56.934940 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40265
I20260929 16:25:56.934959 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40265 every 8 connection(s)
I20260929 16:25:56.935943 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:25:56.944134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 1024
I20260929 16:25:56.944288 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:25:56.944556 392 external_mini_cluster.cc:1428] Setting key 86a1cfce2071d2d85c81de555a0f31c7
Sep 29 16:25:56 dist-test-slave-vq9v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:25:56.953495 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:56.938062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57883 (local address 127.0.98.62:37559)
0929 16:25:56.938348 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:56.938354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:56.939266 (+ 912us) server_negotiation.cc:407] Connection header received
0929 16:25:56.940820 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:56.940823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:56.940877 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:56.940982 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:56.942442 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:56.943023 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:56.944250 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:56.944542 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:56.944666 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:56.948145 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:56.948149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:56.948152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:56.948188 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:56.950295 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:56.951236 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:56.951242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:56.951247 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:56.951354 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:56.951918 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:56.951921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:56.951922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:56.952133 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:56.952236 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:56.953115 (+ 879us) server_negotiation.cc:299] Negotiation successful
0929 16:25:56.953345 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":92,"threads_started":1}
I20260929 16:25:56.954766 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:56.938505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37559 (local address 127.0.98.1:57883)
0929 16:25:56.939076 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:56.939148 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:56.940563 (+ 1415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:56.941157 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:56.941165 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:56.941660 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:56.942248 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:56.942264 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:56.943153 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:56.943156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:56.944097 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:56.944108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:56.944718 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:56.944722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:56.945009 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:56.945769 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:56.945789 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:56.947934 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:56.950541 (+ 2607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:56.950550 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:56.950575 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:56.951100 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:56.951500 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:56.951502 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:56.951504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:56.951767 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:56.952272 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:56.952282 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:56.952710 (+ 428us) client_negotiation.cc:769] Sending connection context
0929 16:25:56.953067 (+ 357us) client_negotiation.cc:240] Negotiation successful
0929 16:25:56.953529 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 16:25:56.956230 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37559
I20260929 16:25:56.956526 1153 heartbeater.cc:464] Registering TS with master...
I20260929 16:25:56.957119 1153 heartbeater.cc:511] Master 127.0.98.62:37559 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 16:25:56.958925 961 ts_manager.cc:194] Registered new tserver with Master: d9ff4842c7494a76bb16c9a7aed6ae6d (127.0.98.1:40265)
I20260929 16:25:56.960111 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 16:25:56.977088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37559
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:25:57.094533 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:25:57.094889 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:25:57.094985 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:25:57.099351 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:25:57.099454 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:25:57.099522 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:25:57.105064 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37559
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:25:57.106189 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:25:57.107197 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:25:57.117420 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:57.117409 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:57.117405 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:57.117766 1159 server_base.cc:1034] running on GCE node
I20260929 16:25:57.118384 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:25:57.119189 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:25:57.120424 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790699157120408 us; error 30 us; skew 500 ppm
Sep 29 16:25:57 dist-test-slave-vq9v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:25:57.123848 1159 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:25:57.124845 1159 webserver.cc:460] Webserver started at http://127.0.98.2:38637/ using document root <none> and password file <none>
I20260929 16:25:57.125551 1159 fs_manager.cc:359] Metadata directory not provided
I20260929 16:25:57.125654 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:25:57.125908 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:25:57.128100 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "016b2ddaf002475cb3f7bf2c9cecfd3d"
format_stamp: "Formatted at 2026-09-29 16:25:57 on dist-test-slave-vq9v"
server_key: "28c08563c3579f166083dc49bddb80e0"
server_key_iv: "ec029df142fe99bfe64bb11739650082"
server_key_version: "encryptionkey@0"
I20260929 16:25:57.128731 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "016b2ddaf002475cb3f7bf2c9cecfd3d"
format_stamp: "Formatted at 2026-09-29 16:25:57 on dist-test-slave-vq9v"
server_key: "28c08563c3579f166083dc49bddb80e0"
server_key_iv: "ec029df142fe99bfe64bb11739650082"
server_key_version: "encryptionkey@0"
I20260929 16:25:57.132993 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:25:57.135790 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:57.137140 1159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:25:57.137336 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "016b2ddaf002475cb3f7bf2c9cecfd3d"
format_stamp: "Formatted at 2026-09-29 16:25:57 on dist-test-slave-vq9v"
server_key: "28c08563c3579f166083dc49bddb80e0"
server_key_iv: "ec029df142fe99bfe64bb11739650082"
server_key_version: "encryptionkey@0"
I20260929 16:25:57.137475 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:25:57.163400 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:25:57.167543 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:25:57.167757 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:25:57.168473 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:25:57.169611 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:25:57.169682 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:57.169749 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:25:57.169770 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:57.181037 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41247
I20260929 16:25:57.181861 1287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41247 every 8 connection(s)
I20260929 16:25:57.182392 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:25:57.184278 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 1159
I20260929 16:25:57.184414 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:25:57.184706 392 external_mini_cluster.cc:1428] Setting key 02eaaf49e97db53c4aa9f66397f1aaca
Sep 29 16:25:57 dist-test-slave-vq9v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 16:25:57.200297 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:57.184602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46213 (local address 127.0.98.62:37559)
0929 16:25:57.184754 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:57.184760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:57.185906 (+ 1146us) server_negotiation.cc:407] Connection header received
0929 16:25:57.187996 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:57.188003 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:57.188079 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:57.188139 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:57.190004 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.190878 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.192198 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.192462 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.192584 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:57.195941 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:57.195945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:57.195947 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:57.195976 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:57.197710 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:57.198493 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:57.198496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:57.198498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:57.198577 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:57.199038 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:57.199040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:57.199042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:57.199235 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:57.199311 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:57.200043 (+ 732us) server_negotiation.cc:299] Negotiation successful
0929 16:25:57.200160 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:25:57.201403 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:57.185159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37559 (local address 127.0.98.2:46213)
0929 16:25:57.185755 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:57.185808 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:57.187645 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:57.188615 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:57.188623 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:57.189004 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:57.189794 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:57.189818 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.191026 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.191029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:57.192050 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:57.192061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.192631 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.192635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:57.192831 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:57.193528 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:57.193555 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:57.195720 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:57.197932 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:57.197940 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:57.197963 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:57.198360 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:57.198680 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:57.198683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:57.198686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:57.198920 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:57.199344 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:57.199352 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:57.199741 (+ 389us) client_negotiation.cc:769] Sending connection context
0929 16:25:57.199969 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 16:25:57.200358 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":139,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 16:25:57.203063 1288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37559
I20260929 16:25:57.203377 1288 heartbeater.cc:464] Registering TS with master...
I20260929 16:25:57.203960 1288 heartbeater.cc:511] Master 127.0.98.62:37559 requested a full tablet report, sending...
I20260929 16:25:57.205538 961 ts_manager.cc:194] Registered new tserver with Master: 016b2ddaf002475cb3f7bf2c9cecfd3d (127.0.98.2:41247)
I20260929 16:25:57.206192 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 16:25:57.225518 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37559
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:25:57.345583 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:25:57.345921 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:25:57.346005 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:25:57.350234 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:25:57.350351 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:25:57.350453 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:25:57.356084 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37559
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:25:57.357282 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:25:57.358359 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:25:57.368508 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:57.368502 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:57.368636 1293 server_base.cc:1034] running on GCE node
W20260929 16:25:57.368956 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:57.369477 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:25:57.370222 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:25:57.371441 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790699157371418 us; error 46 us; skew 500 ppm
Sep 29 16:25:57 dist-test-slave-vq9v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:25:57.374488 1293 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:25:57.375429 1293 webserver.cc:460] Webserver started at http://127.0.98.3:33043/ using document root <none> and password file <none>
I20260929 16:25:57.375979 1293 fs_manager.cc:359] Metadata directory not provided
I20260929 16:25:57.376041 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:25:57.376212 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:25:57.378397 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "32d8c2cf89f94ee79391bbe61e41c385"
format_stamp: "Formatted at 2026-09-29 16:25:57 on dist-test-slave-vq9v"
server_key: "4065b12dcab319f448d7713425062689"
server_key_iv: "ca6ee955e67ea09ad542520077789538"
server_key_version: "encryptionkey@0"
I20260929 16:25:57.378923 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "32d8c2cf89f94ee79391bbe61e41c385"
format_stamp: "Formatted at 2026-09-29 16:25:57 on dist-test-slave-vq9v"
server_key: "4065b12dcab319f448d7713425062689"
server_key_iv: "ca6ee955e67ea09ad542520077789538"
server_key_version: "encryptionkey@0"
I20260929 16:25:57.382747 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:25:57.385578 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:57.386955 1293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:25:57.387065 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "32d8c2cf89f94ee79391bbe61e41c385"
format_stamp: "Formatted at 2026-09-29 16:25:57 on dist-test-slave-vq9v"
server_key: "4065b12dcab319f448d7713425062689"
server_key_iv: "ca6ee955e67ea09ad542520077789538"
server_key_version: "encryptionkey@0"
I20260929 16:25:57.387223 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:25:57.409037 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:25:57.412985 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:25:57.413203 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:25:57.413910 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:25:57.414776 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:25:57.414824 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:57.414872 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:25:57.414928 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:57.424613 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35591
I20260929 16:25:57.424638 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35591 every 8 connection(s)
I20260929 16:25:57.425596 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:25:57.433123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 1293
I20260929 16:25:57.433310 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:25:57.433636 392 external_mini_cluster.cc:1428] Setting key 6a4f9b07e09933de62fd5b1e0f2c0ca3
Sep 29 16:25:57 dist-test-slave-vq9v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:25:57.442077 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:57.427628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39881 (local address 127.0.98.62:37559)
0929 16:25:57.427868 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:57.427872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:57.428934 (+ 1062us) server_negotiation.cc:407] Connection header received
0929 16:25:57.430419 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:57.430423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:57.430488 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:57.430616 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:57.431988 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.432552 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.434322 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.434655 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.434787 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:57.437945 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:57.437948 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:57.437951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:57.437999 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:57.439675 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:57.440271 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:57.440275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:57.440277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:57.440366 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:57.440759 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:57.440762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:57.440764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:57.440983 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:57.441096 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:57.441764 (+ 668us) server_negotiation.cc:299] Negotiation successful
0929 16:25:57.441909 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 16:25:57.443032 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:57.428154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37559 (local address 127.0.98.3:39881)
0929 16:25:57.428680 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:57.428778 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:57.430150 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:57.430816 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:57.430825 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:57.431239 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:57.431805 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:57.431822 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.432709 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.432726 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:57.434168 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:57.434180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.434850 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.434855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:57.435055 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:57.435776 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:57.435796 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:57.437663 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:57.439816 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:57.439824 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:57.439844 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:57.440147 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:57.440470 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:57.440473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:57.440475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:57.440652 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:57.441120 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:57.441129 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:57.441473 (+ 344us) client_negotiation.cc:769] Sending connection context
0929 16:25:57.441711 (+ 238us) client_negotiation.cc:240] Negotiation successful
0929 16:25:57.442083 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":124,"threads_started":1}
I20260929 16:25:57.444345 1422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37559
I20260929 16:25:57.444577 1422 heartbeater.cc:464] Registering TS with master...
I20260929 16:25:57.445061 1422 heartbeater.cc:511] Master 127.0.98.62:37559 requested a full tablet report, sending...
I20260929 16:25:57.446305 961 ts_manager.cc:194] Registered new tserver with Master: 32d8c2cf89f94ee79391bbe61e41c385 (127.0.98.3:35591)
I20260929 16:25:57.447012 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39881
I20260929 16:25:57.448088 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:25:57.460018 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:57.452182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.0.98.62:37559)
0929 16:25:57.452353 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:57.452357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:57.452373 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:25:57.452414 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:57.452416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:57.452460 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:57.452552 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:57.453380 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.453930 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.454813 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.455052 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.455165 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:57.456700 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:57.456704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:57.456706 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:57.456731 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:57.458336 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:57.458785 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:57.458788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:57.458789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:57.458922 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:57.459178 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:57.459180 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:57.459181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:57.459426 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:57.459533 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:57.459746 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 16:25:57.459894 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:25:57.465883 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:25:57.468997 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:25:57.485785 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:57.478780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40265 (local address 127.0.0.1:41104)
0929 16:25:57.479780 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:57.479830 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:57.479974 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:57.480540 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:57.480546 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:57.480567 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:25:57.480720 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:57.480728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.482223 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.482230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:57.483765 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:57.483775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.485067 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.485071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:57.485287 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:57.485355 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 16:25:57.485471 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 16:25:57.485621 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":105,"thread_start_us":95,"threads_started":1}
I20260929 16:25:57.485839 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:57.479558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49804 (local address 127.0.98.2:41247)
0929 16:25:57.480755 (+ 1197us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:57.480768 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:57.480791 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:25:57.480859 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:57.480865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:57.481011 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:57.481167 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:25:57.481402 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.482152 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.483907 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.484739 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.484846 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:57.485001 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:57.485596 (+ 595us) server_negotiation.cc:299] Negotiation successful
0929 16:25:57.485696 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1099,"thread_start_us":133,"threads_started":1}
I20260929 16:25:57.486464 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:57.478923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41247 (local address 127.0.0.1:49804)
0929 16:25:57.480289 (+ 1366us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:57.480303 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:57.480394 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:57.481146 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:57.481149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:57.481165 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:25:57.481283 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:57.481289 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.482299 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.482302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:57.483772 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:57.483778 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.485067 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.485071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:57.485291 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:57.485353 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 16:25:57.485471 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 16:25:57.486293 (+ 822us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1267,"mutex_wait_us":807,"thread_start_us":90,"threads_started":1}
I20260929 16:25:57.486693 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:57.479557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41104 (local address 127.0.98.1:40265)
0929 16:25:57.479873 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:57.479881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:57.479909 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:25:57.480163 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:57.480170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:57.480401 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:57.480558 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:25:57.480854 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.482104 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.483915 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.484739 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.484846 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:57.485001 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:57.486329 (+ 1328us) server_negotiation.cc:299] Negotiation successful
0929 16:25:57.486449 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":140,"threads_started":1}
I20260929 16:25:57.487622 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:57.478729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35591 (local address 127.0.0.1:39826)
0929 16:25:57.479917 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:57.479934 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:57.480037 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:57.480899 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:57.480904 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:57.480926 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:25:57.481023 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:57.481030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.482495 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.482501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:57.485625 (+ 3124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:57.485638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.487133 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.487140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:57.487263 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:57.487291 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 16:25:57.487369 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 16:25:57.487447 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":138,"threads_started":1}
I20260929 16:25:57.488191 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:57.478864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39826 (local address 127.0.98.3:35591)
0929 16:25:57.480267 (+ 1403us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:57.480278 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:57.480299 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:25:57.480371 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:57.480379 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:57.480577 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:57.480731 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:25:57.481169 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.482319 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.485801 (+ 3482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.486793 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.486907 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:57.487066 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:57.487925 (+ 859us) server_negotiation.cc:299] Negotiation successful
0929 16:25:57.488030 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1306,"thread_start_us":138,"threads_started":1}
I20260929 16:25:57.489661 1087 tablet_service.cc:1467] Processing CreateTablet for tablet f98d59e1b74f4773bc7b0586b0db07f2 (DEFAULT_TABLE table=test-table [id=6f19d1ad7dae4fac95092a9213bfe236]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:25:57.489673 1222 tablet_service.cc:1467] Processing CreateTablet for tablet f98d59e1b74f4773bc7b0586b0db07f2 (DEFAULT_TABLE table=test-table [id=6f19d1ad7dae4fac95092a9213bfe236]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:25:57.490796 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f98d59e1b74f4773bc7b0586b0db07f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:25:57.490804 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f98d59e1b74f4773bc7b0586b0db07f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:25:57.491042 1356 tablet_service.cc:1467] Processing CreateTablet for tablet f98d59e1b74f4773bc7b0586b0db07f2 (DEFAULT_TABLE table=test-table [id=6f19d1ad7dae4fac95092a9213bfe236]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:25:57.491994 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f98d59e1b74f4773bc7b0586b0db07f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:25:57.498458 1440 tablet_bootstrap.cc:492] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d: Bootstrap starting.
I20260929 16:25:57.499881 1441 tablet_bootstrap.cc:492] T f98d59e1b74f4773bc7b0586b0db07f2 P 016b2ddaf002475cb3f7bf2c9cecfd3d: Bootstrap starting.
I20260929 16:25:57.500958 1440 tablet_bootstrap.cc:654] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d: Neither blocks nor log segments found. Creating new log.
I20260929 16:25:57.501557 1442 tablet_bootstrap.cc:492] T f98d59e1b74f4773bc7b0586b0db07f2 P 32d8c2cf89f94ee79391bbe61e41c385: Bootstrap starting.
I20260929 16:25:57.501801 1441 tablet_bootstrap.cc:654] T f98d59e1b74f4773bc7b0586b0db07f2 P 016b2ddaf002475cb3f7bf2c9cecfd3d: Neither blocks nor log segments found. Creating new log.
I20260929 16:25:57.501955 1440 log.cc:824] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:25:57.503142 1441 log.cc:824] T f98d59e1b74f4773bc7b0586b0db07f2 P 016b2ddaf002475cb3f7bf2c9cecfd3d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:25:57.504017 1442 tablet_bootstrap.cc:654] T f98d59e1b74f4773bc7b0586b0db07f2 P 32d8c2cf89f94ee79391bbe61e41c385: Neither blocks nor log segments found. Creating new log.
I20260929 16:25:57.505299 1442 log.cc:824] T f98d59e1b74f4773bc7b0586b0db07f2 P 32d8c2cf89f94ee79391bbe61e41c385: Log is configured to *not* fsync() on all Append() calls
I20260929 16:25:57.505554 1441 tablet_bootstrap.cc:492] T f98d59e1b74f4773bc7b0586b0db07f2 P 016b2ddaf002475cb3f7bf2c9cecfd3d: No bootstrap required, opened a new log
I20260929 16:25:57.505805 1441 ts_tablet_manager.cc:1397] T f98d59e1b74f4773bc7b0586b0db07f2 P 016b2ddaf002475cb3f7bf2c9cecfd3d: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:25:57.507154 1440 tablet_bootstrap.cc:492] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d: No bootstrap required, opened a new log
I20260929 16:25:57.507191 1442 tablet_bootstrap.cc:492] T f98d59e1b74f4773bc7b0586b0db07f2 P 32d8c2cf89f94ee79391bbe61e41c385: No bootstrap required, opened a new log
I20260929 16:25:57.507351 1440 ts_tablet_manager.cc:1397] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260929 16:25:57.507411 1442 ts_tablet_manager.cc:1397] T f98d59e1b74f4773bc7b0586b0db07f2 P 32d8c2cf89f94ee79391bbe61e41c385: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:25:57.510123 1441 raft_consensus.cc:348] T f98d59e1b74f4773bc7b0586b0db07f2 P 016b2ddaf002475cb3f7bf2c9cecfd3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d8c2cf89f94ee79391bbe61e41c385" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35591 } } peers { permanent_uuid: "d9ff4842c7494a76bb16c9a7aed6ae6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40265 } } peers { permanent_uuid: "016b2ddaf002475cb3f7bf2c9cecfd3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41247 } }
I20260929 16:25:57.510452 1441 raft_consensus.cc:374] T f98d59e1b74f4773bc7b0586b0db07f2 P 016b2ddaf002475cb3f7bf2c9cecfd3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:25:57.510512 1441 raft_consensus.cc:729] T f98d59e1b74f4773bc7b0586b0db07f2 P 016b2ddaf002475cb3f7bf2c9cecfd3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016b2ddaf002475cb3f7bf2c9cecfd3d, State: Initialized, Role: FOLLOWER
I20260929 16:25:57.511008 1441 consensus_queue.cc:260] T f98d59e1b74f4773bc7b0586b0db07f2 P 016b2ddaf002475cb3f7bf2c9cecfd3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d8c2cf89f94ee79391bbe61e41c385" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35591 } } peers { permanent_uuid: "d9ff4842c7494a76bb16c9a7aed6ae6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40265 } } peers { permanent_uuid: "016b2ddaf002475cb3f7bf2c9cecfd3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41247 } }
I20260929 16:25:57.511232 1442 raft_consensus.cc:348] T f98d59e1b74f4773bc7b0586b0db07f2 P 32d8c2cf89f94ee79391bbe61e41c385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d8c2cf89f94ee79391bbe61e41c385" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35591 } } peers { permanent_uuid: "d9ff4842c7494a76bb16c9a7aed6ae6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40265 } } peers { permanent_uuid: "016b2ddaf002475cb3f7bf2c9cecfd3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41247 } }
I20260929 16:25:57.511548 1442 raft_consensus.cc:374] T f98d59e1b74f4773bc7b0586b0db07f2 P 32d8c2cf89f94ee79391bbe61e41c385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:25:57.511629 1442 raft_consensus.cc:729] T f98d59e1b74f4773bc7b0586b0db07f2 P 32d8c2cf89f94ee79391bbe61e41c385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d8c2cf89f94ee79391bbe61e41c385, State: Initialized, Role: FOLLOWER
I20260929 16:25:57.511951 1288 heartbeater.cc:503] Master 127.0.98.62:37559 was elected leader, sending a full tablet report...
I20260929 16:25:57.511770 1440 raft_consensus.cc:348] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d8c2cf89f94ee79391bbe61e41c385" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35591 } } peers { permanent_uuid: "d9ff4842c7494a76bb16c9a7aed6ae6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40265 } } peers { permanent_uuid: "016b2ddaf002475cb3f7bf2c9cecfd3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41247 } }
I20260929 16:25:57.512032 1440 raft_consensus.cc:374] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:25:57.512089 1440 raft_consensus.cc:729] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ff4842c7494a76bb16c9a7aed6ae6d, State: Initialized, Role: FOLLOWER
I20260929 16:25:57.512074 1442 consensus_queue.cc:260] T f98d59e1b74f4773bc7b0586b0db07f2 P 32d8c2cf89f94ee79391bbe61e41c385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d8c2cf89f94ee79391bbe61e41c385" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35591 } } peers { permanent_uuid: "d9ff4842c7494a76bb16c9a7aed6ae6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40265 } } peers { permanent_uuid: "016b2ddaf002475cb3f7bf2c9cecfd3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41247 } }
I20260929 16:25:57.512508 1440 consensus_queue.cc:260] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d8c2cf89f94ee79391bbe61e41c385" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35591 } } peers { permanent_uuid: "d9ff4842c7494a76bb16c9a7aed6ae6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40265 } } peers { permanent_uuid: "016b2ddaf002475cb3f7bf2c9cecfd3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41247 } }
I20260929 16:25:57.512691 1422 heartbeater.cc:503] Master 127.0.98.62:37559 was elected leader, sending a full tablet report...
I20260929 16:25:57.513500 1441 ts_tablet_manager.cc:1428] T f98d59e1b74f4773bc7b0586b0db07f2 P 016b2ddaf002475cb3f7bf2c9cecfd3d: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260929 16:25:57.513500 1440 ts_tablet_manager.cc:1428] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:25:57.514423 1442 ts_tablet_manager.cc:1428] T f98d59e1b74f4773bc7b0586b0db07f2 P 32d8c2cf89f94ee79391bbe61e41c385: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 16:25:57.514787 1153 heartbeater.cc:503] Master 127.0.98.62:37559 was elected leader, sending a full tablet report...
I20260929 16:25:57.535820 1448 raft_consensus.cc:482] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:25:57.536065 1448 raft_consensus.cc:504] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d8c2cf89f94ee79391bbe61e41c385" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35591 } } peers { permanent_uuid: "d9ff4842c7494a76bb16c9a7aed6ae6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40265 } } peers { permanent_uuid: "016b2ddaf002475cb3f7bf2c9cecfd3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41247 } }
I20260929 16:25:57.537420 1448 leader_election.cc:288] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32d8c2cf89f94ee79391bbe61e41c385 (127.0.98.3:35591), 016b2ddaf002475cb3f7bf2c9cecfd3d (127.0.98.2:41247)
I20260929 16:25:57.542085 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:57.537698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44329 (local address 127.0.98.3:35591)
0929 16:25:57.537847 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:57.537863 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:57.538126 (+ 263us) server_negotiation.cc:407] Connection header received
0929 16:25:57.538226 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:57.538232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:57.538289 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:57.538370 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:25:57.538673 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.539285 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.540827 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.541498 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.541607 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:57.541673 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:57.541866 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 16:25:57.541938 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:25:57.542203 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:57.537626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35591 (local address 127.0.98.1:44329)
0929 16:25:57.538022 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:57.538034 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:57.538135 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:57.538395 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:57.538398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:57.538415 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:25:57.538561 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:57.538568 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.539388 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.539391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:57.540707 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:57.540713 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.541626 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.541629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:57.541722 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:57.541749 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:25:57.541830 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 16:25:57.541906 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":126,"thread_start_us":118,"threads_started":1}
I20260929 16:25:57.542629 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:57.537848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33273 (local address 127.0.98.2:41247)
0929 16:25:57.537976 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:57.537989 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:57.538267 (+ 278us) server_negotiation.cc:407] Connection header received
0929 16:25:57.538414 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:57.538417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:57.538468 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:57.538551 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:25:57.538823 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.539460 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.541287 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.541924 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.542028 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:57.542084 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:57.542438 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 16:25:57.542521 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:25:57.542886 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f98d59e1b74f4773bc7b0586b0db07f2" candidate_uuid: "d9ff4842c7494a76bb16c9a7aed6ae6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d8c2cf89f94ee79391bbe61e41c385" is_pre_election: true
I20260929 16:25:57.543004 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:57.537698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41247 (local address 127.0.98.1:33273)
0929 16:25:57.538158 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:57.538173 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:57.538274 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:57.538568 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:57.538570 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:57.538584 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:25:57.538718 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:57.538724 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.539570 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.539572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:57.540700 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:57.540711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.542149 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.542152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:57.542254 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:57.542284 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:25:57.542777 (+ 493us) client_negotiation.cc:240] Negotiation successful
0929 16:25:57.542857 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":83,"thread_start_us":130,"threads_started":1}
I20260929 16:25:57.543208 1376 raft_consensus.cc:2454] T f98d59e1b74f4773bc7b0586b0db07f2 P 32d8c2cf89f94ee79391bbe61e41c385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9ff4842c7494a76bb16c9a7aed6ae6d in term 0.
I20260929 16:25:57.543604 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f98d59e1b74f4773bc7b0586b0db07f2" candidate_uuid: "d9ff4842c7494a76bb16c9a7aed6ae6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016b2ddaf002475cb3f7bf2c9cecfd3d" is_pre_election: true
I20260929 16:25:57.543726 1042 leader_election.cc:302] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32d8c2cf89f94ee79391bbe61e41c385, d9ff4842c7494a76bb16c9a7aed6ae6d; no voters:
I20260929 16:25:57.543871 1242 raft_consensus.cc:2454] T f98d59e1b74f4773bc7b0586b0db07f2 P 016b2ddaf002475cb3f7bf2c9cecfd3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9ff4842c7494a76bb16c9a7aed6ae6d in term 0.
I20260929 16:25:57.544071 1448 raft_consensus.cc:2781] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:25:57.544171 1448 raft_consensus.cc:482] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:25:57.544231 1448 raft_consensus.cc:3037] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:25:57.545302 1448 raft_consensus.cc:504] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d8c2cf89f94ee79391bbe61e41c385" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35591 } } peers { permanent_uuid: "d9ff4842c7494a76bb16c9a7aed6ae6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40265 } } peers { permanent_uuid: "016b2ddaf002475cb3f7bf2c9cecfd3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41247 } }
I20260929 16:25:57.545701 1448 leader_election.cc:288] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d [CANDIDATE]: Term 1 election: Requested vote from peers 32d8c2cf89f94ee79391bbe61e41c385 (127.0.98.3:35591), 016b2ddaf002475cb3f7bf2c9cecfd3d (127.0.98.2:41247)
I20260929 16:25:57.546058 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f98d59e1b74f4773bc7b0586b0db07f2" candidate_uuid: "d9ff4842c7494a76bb16c9a7aed6ae6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "016b2ddaf002475cb3f7bf2c9cecfd3d"
I20260929 16:25:57.546080 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f98d59e1b74f4773bc7b0586b0db07f2" candidate_uuid: "d9ff4842c7494a76bb16c9a7aed6ae6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d8c2cf89f94ee79391bbe61e41c385"
I20260929 16:25:57.546171 1242 raft_consensus.cc:3037] T f98d59e1b74f4773bc7b0586b0db07f2 P 016b2ddaf002475cb3f7bf2c9cecfd3d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:25:57.546206 1376 raft_consensus.cc:3037] T f98d59e1b74f4773bc7b0586b0db07f2 P 32d8c2cf89f94ee79391bbe61e41c385 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:25:57.547260 1242 raft_consensus.cc:2454] T f98d59e1b74f4773bc7b0586b0db07f2 P 016b2ddaf002475cb3f7bf2c9cecfd3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9ff4842c7494a76bb16c9a7aed6ae6d in term 1.
I20260929 16:25:57.547261 1376 raft_consensus.cc:2454] T f98d59e1b74f4773bc7b0586b0db07f2 P 32d8c2cf89f94ee79391bbe61e41c385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9ff4842c7494a76bb16c9a7aed6ae6d in term 1.
I20260929 16:25:57.547587 1040 leader_election.cc:302] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 016b2ddaf002475cb3f7bf2c9cecfd3d, d9ff4842c7494a76bb16c9a7aed6ae6d; no voters:
I20260929 16:25:57.547837 1448 raft_consensus.cc:2781] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:25:57.548090 1448 raft_consensus.cc:686] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d [term 1 LEADER]: Becoming Leader. State: Replica: d9ff4842c7494a76bb16c9a7aed6ae6d, State: Running, Role: LEADER
I20260929 16:25:57.548429 1448 consensus_queue.cc:237] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d8c2cf89f94ee79391bbe61e41c385" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35591 } } peers { permanent_uuid: "d9ff4842c7494a76bb16c9a7aed6ae6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40265 } } peers { permanent_uuid: "016b2ddaf002475cb3f7bf2c9cecfd3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41247 } }
I20260929 16:25:57.551714 962 catalog_manager.cc:5495] T f98d59e1b74f4773bc7b0586b0db07f2 P d9ff4842c7494a76bb16c9a7aed6ae6d reported cstate change: term changed from 0 to 1, leader changed from <none> to d9ff4842c7494a76bb16c9a7aed6ae6d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d9ff4842c7494a76bb16c9a7aed6ae6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d8c2cf89f94ee79391bbe61e41c385" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9ff4842c7494a76bb16c9a7aed6ae6d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "016b2ddaf002475cb3f7bf2c9cecfd3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41247 } health_report { overall_health: UNKNOWN } } }
Sep 29 16:25:57 dist-test-slave-vq9v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 16:25:57.573052 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:57.559397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41112 (local address 127.0.98.1:40265)
0929 16:25:57.559534 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:57.559538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:57.559794 (+ 256us) server_negotiation.cc:407] Connection header received
0929 16:25:57.559880 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:57.559883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:57.559929 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:57.560006 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:57.560663 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.561350 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.562229 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.562449 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.562536 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:57.569155 (+ 6619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:57.569171 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:57.569186 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:57.569293 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:57.571133 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:57.571689 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:57.571696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:57.571697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:57.571842 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:57.572156 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:57.572160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:57.572161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:57.572439 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:57.572621 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:57.572797 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 16:25:57.572916 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 16:25:57 dist-test-slave-vq9v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699157, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:25:57 dist-test-slave-vq9v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699157, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 16:25:57.602483 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:57.592640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.0.98.1:40265)
0929 16:25:57.592844 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:57.592848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:57.592992 (+ 144us) server_negotiation.cc:407] Connection header received
0929 16:25:57.593077 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:57.593080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:57.593135 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:57.593323 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:57.594110 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.594777 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.595754 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.595996 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.596098 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:57.599164 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:57.599168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:57.599170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:57.599202 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:57.600619 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:57.601118 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:57.601121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:57.601122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:57.601302 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:57.601567 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:57.601569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:57.601570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:57.601747 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:57.601872 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:57.602241 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 16:25:57.602351 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 16:25:57 dist-test-slave-vq9v krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:25:57 dist-test-slave-vq9v krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 16:25:57.631404 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:57.621371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41138 (local address 127.0.98.1:40265)
0929 16:25:57.621588 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:57.621594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:57.621741 (+ 147us) server_negotiation.cc:407] Connection header received
0929 16:25:57.621930 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:57.621934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:57.621995 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:57.622088 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:57.622918 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.623475 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.624402 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:57.624639 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:57.624780 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:57.627707 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:57.627710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:57.627712 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:57.627750 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:57.629511 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:57.630044 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:57.630047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:57.630048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:57.630146 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:57.630485 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:57.630487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:57.630489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:57.630680 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:57.630863 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:57.631136 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 16:25:57.631259 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
W20260929 16:25:57.632938 1067 tablet_service.cc:2908] Rejecting scan request for tablet f98d59e1b74f4773bc7b0586b0db07f2: Uninitialized: clean time has not yet been initialized
I20260929 16:25:57.635161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 1024
I20260929 16:25:57.641712 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 1159
I20260929 16:25:57.647505 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 1293
I20260929 16:25:57.653858 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 930
2026-09-29T16:25:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2894 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:25:57 dist-test-slave-vq9v krb5kdc[1465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:25:57 dist-test-slave-vq9v krb5kdc[1465](info): set up 2 sockets
Sep 29 16:25:57 dist-test-slave-vq9v krb5kdc[1465](info): commencing operation
krb5kdc: starting...
W20260929 16:25:58.733312 392 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:25:58 dist-test-slave-vq9v krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:25:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:25:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:25:58.870000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34879
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:25:58.983618 1478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:25:58.983966 1478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:25:58.984046 1478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:25:58.988121 1478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:25:58.988222 1478 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 16:25:58.988256 1478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:25:58.988299 1478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:25:58.988335 1478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:25:58.988370 1478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:25:58.993848 1478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34879
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:25:58.994976 1478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:25:58.996025 1478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:25:59.005580 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:59.005594 1478 server_base.cc:1034] running on GCE node
W20260929 16:25:59.005573 1484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:59.005580 1483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:59.006610 1478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:25:59.007845 1478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:25:59.009088 1478 hybrid_clock.cc:648] HybridClock initialized: now 1790699159009068 us; error 44 us; skew 500 ppm
Sep 29 16:25:59 dist-test-slave-vq9v krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:25:59.012503 1478 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:25:59.013613 1478 webserver.cc:460] Webserver started at http://127.0.98.62:45721/ using document root <none> and password file <none>
I20260929 16:25:59.014200 1478 fs_manager.cc:359] Metadata directory not provided
I20260929 16:25:59.014295 1478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:25:59.014509 1478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:25:59.016708 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "87d46e4029ef47d2b0890bb954a13ad7"
format_stamp: "Formatted at 2026-09-29 16:25:59 on dist-test-slave-vq9v"
server_key: "36b91986ffb0c2a4d2a9b03866ad6107"
server_key_iv: "87306ded244181feb3bd508e0013f375"
server_key_version: "encryptionkey@0"
I20260929 16:25:59.017385 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "87d46e4029ef47d2b0890bb954a13ad7"
format_stamp: "Formatted at 2026-09-29 16:25:59 on dist-test-slave-vq9v"
server_key: "36b91986ffb0c2a4d2a9b03866ad6107"
server_key_iv: "87306ded244181feb3bd508e0013f375"
server_key_version: "encryptionkey@0"
I20260929 16:25:59.021467 1478 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:25:59.024029 1493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:59.025494 1478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:25:59.025638 1478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "87d46e4029ef47d2b0890bb954a13ad7"
format_stamp: "Formatted at 2026-09-29 16:25:59 on dist-test-slave-vq9v"
server_key: "36b91986ffb0c2a4d2a9b03866ad6107"
server_key_iv: "87306ded244181feb3bd508e0013f375"
server_key_version: "encryptionkey@0"
I20260929 16:25:59.025763 1478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:25:59.054960 1478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:25:59.058843 1478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:25:59.059100 1478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:25:59.067513 1478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34879
I20260929 16:25:59.067528 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34879 every 8 connection(s)
I20260929 16:25:59.068527 1478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:25:59.071449 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:25:59.077565 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 1478
I20260929 16:25:59.077548 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7: Bootstrap starting.
I20260929 16:25:59.077677 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:25:59.077939 392 external_mini_cluster.cc:1428] Setting key 1c9333acd59ae88ef8839a124c874b2d
I20260929 16:25:59.080322 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7: Neither blocks nor log segments found. Creating new log.
I20260929 16:25:59.081110 1556 log.cc:824] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7: Log is configured to *not* fsync() on all Append() calls
I20260929 16:25:59.083343 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7: No bootstrap required, opened a new log
Sep 29 16:25:59 dist-test-slave-vq9v krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:25:59.087325 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d46e4029ef47d2b0890bb954a13ad7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } }
I20260929 16:25:59.087673 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:25:59.087756 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d46e4029ef47d2b0890bb954a13ad7, State: Initialized, Role: FOLLOWER
I20260929 16:25:59.088218 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d46e4029ef47d2b0890bb954a13ad7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } }
I20260929 16:25:59.088379 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:25:59.088470 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:25:59.088570 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:25:59.089598 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d46e4029ef47d2b0890bb954a13ad7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } }
I20260929 16:25:59.089954 1556 leader_election.cc:302] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87d46e4029ef47d2b0890bb954a13ad7; no voters:
I20260929 16:25:59.090309 1556 leader_election.cc:288] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:25:59.090395 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:25:59.090575 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7 [term 1 LEADER]: Becoming Leader. State: Replica: 87d46e4029ef47d2b0890bb954a13ad7, State: Running, Role: LEADER
I20260929 16:25:59.090869 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d46e4029ef47d2b0890bb954a13ad7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } }
I20260929 16:25:59.091634 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:25:59.092041 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87d46e4029ef47d2b0890bb954a13ad7. Latest consensus state: current_term: 1 leader_uuid: "87d46e4029ef47d2b0890bb954a13ad7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d46e4029ef47d2b0890bb954a13ad7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } }
I20260929 16:25:59.092149 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7 [sys.catalog]: This master's current role is: LEADER
I20260929 16:25:59.092833 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87d46e4029ef47d2b0890bb954a13ad7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87d46e4029ef47d2b0890bb954a13ad7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } }
I20260929 16:25:59.092947 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7 [sys.catalog]: This master's current role is: LEADER
I20260929 16:25:59.093227 1565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:25:59.095208 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:59.079449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36166 (local address 127.0.98.62:34879)
0929 16:25:59.079890 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:59.079906 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:59.079943 (+ 37us) server_negotiation.cc:407] Connection header received
0929 16:25:59.080698 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:59.080719 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:59.081083 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:59.081493 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:59.082354 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.083387 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.084318 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.084810 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.084986 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:59.088237 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:59.088243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:59.088260 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:59.088291 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:59.090710 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:59.091306 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:59.091311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:59.091316 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:59.091437 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:59.091862 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:59.091865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:59.091869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:59.092596 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:59.093711 (+ 1115us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:59.093883 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 16:25:59.094225 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":150,"threads_started":1}
I20260929 16:25:59.096547 1565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:25:59.103099 1565 catalog_manager.cc:1347] Generated new cluster ID: 69cc98f520b5428f966315629110717d
I20260929 16:25:59.103209 1565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:25:59.112428 1565 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:25:59.113579 1565 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:25:59.123687 1565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7: Generated new TSK 0
I20260929 16:25:59.124607 1565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 16:25:59.165768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34879
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:25:59.280786 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:25:59.281164 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:25:59.281286 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:25:59.285605 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:25:59.285710 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:25:59.285784 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:25:59.285873 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:25:59.291597 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:25:59.292941 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:25:59.294159 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:25:59.304312 1582 server_base.cc:1034] running on GCE node
W20260929 16:25:59.304260 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:59.304270 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:59.304283 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:59.305040 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:25:59.305780 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:25:59.306995 1582 hybrid_clock.cc:648] HybridClock initialized: now 1790699159306977 us; error 34 us; skew 500 ppm
Sep 29 16:25:59 dist-test-slave-vq9v krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:25:59.310067 1582 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:25:59.311097 1582 webserver.cc:460] Webserver started at http://127.0.98.1:42919/ using document root <none> and password file <none>
I20260929 16:25:59.311697 1582 fs_manager.cc:359] Metadata directory not provided
I20260929 16:25:59.311793 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:25:59.312052 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:25:59.314247 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "4f5fa3284cff4be68ddf362d78991ce0"
format_stamp: "Formatted at 2026-09-29 16:25:59 on dist-test-slave-vq9v"
server_key: "75ec4f1838b996d33d2591579ecd98c6"
server_key_iv: "b316a0b5ad677f2da838451d954b2143"
server_key_version: "encryptionkey@0"
I20260929 16:25:59.314831 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4f5fa3284cff4be68ddf362d78991ce0"
format_stamp: "Formatted at 2026-09-29 16:25:59 on dist-test-slave-vq9v"
server_key: "75ec4f1838b996d33d2591579ecd98c6"
server_key_iv: "b316a0b5ad677f2da838451d954b2143"
server_key_version: "encryptionkey@0"
I20260929 16:25:59.318899 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:25:59.321789 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:59.323374 1582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:25:59.323559 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "4f5fa3284cff4be68ddf362d78991ce0"
format_stamp: "Formatted at 2026-09-29 16:25:59 on dist-test-slave-vq9v"
server_key: "75ec4f1838b996d33d2591579ecd98c6"
server_key_iv: "b316a0b5ad677f2da838451d954b2143"
server_key_version: "encryptionkey@0"
I20260929 16:25:59.323704 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:25:59.343210 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:25:59.346794 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:25:59.347023 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:25:59.348521 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:25:59.348601 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:59.348675 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:25:59.348726 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:25:59 dist-test-slave-vq9v krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:25:59.360383 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40661
I20260929 16:25:59.360535 1712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40661 every 8 connection(s)
I20260929 16:25:59.361601 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:25:59.362911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 1582
I20260929 16:25:59.363044 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:25:59.363332 392 external_mini_cluster.cc:1428] Setting key 5fc665321293bcf9170fbb7db4e7b2ec
I20260929 16:25:59.364035 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:59.349527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56833 (local address 127.0.98.62:34879)
0929 16:25:59.349660 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:59.349663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:59.350798 (+ 1135us) server_negotiation.cc:407] Connection header received
0929 16:25:59.351989 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:59.351995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:59.352068 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:59.352175 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:59.353814 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.354350 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.355573 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.355817 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.355951 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:59.359704 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:59.359707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:59.359710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:59.359739 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:59.361462 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:59.362120 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:59.362125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:59.362126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:59.362221 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:59.362632 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:59.362635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:59.362637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:59.362896 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:59.363014 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:59.363701 (+ 687us) server_negotiation.cc:299] Negotiation successful
0929 16:25:59.363842 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:25:59.364969 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:59.350099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34879 (local address 127.0.98.1:56833)
0929 16:25:59.350607 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:59.350662 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:59.351730 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:59.352437 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:59.352455 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:59.352961 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:59.353640 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:59.353665 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.354492 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.354496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:59.355426 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:59.355435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.355996 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.356001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:59.356217 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:59.356981 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:59.356997 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:59.359470 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:59.361597 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:59.361603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:59.361620 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:59.361988 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:59.362339 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:59.362342 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:59.362344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:59.362527 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:59.363103 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:59.363113 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:59.363420 (+ 307us) client_negotiation.cc:769] Sending connection context
0929 16:25:59.363623 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 16:25:59.364007 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":125,"threads_started":1}
I20260929 16:25:59.366431 1713 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34879
I20260929 16:25:59.366619 1713 heartbeater.cc:464] Registering TS with master...
I20260929 16:25:59.367300 1713 heartbeater.cc:511] Master 127.0.98.62:34879 requested a full tablet report, sending...
I20260929 16:25:59.369145 1509 ts_manager.cc:194] Registered new tserver with Master: 4f5fa3284cff4be68ddf362d78991ce0 (127.0.98.1:40661)
I20260929 16:25:59.371172 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56833
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 16:25:59.377297 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:59.368932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41009 (local address 127.0.98.62:34879)
0929 16:25:59.369058 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:59.369063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:59.369312 (+ 249us) server_negotiation.cc:407] Connection header received
0929 16:25:59.369514 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:59.369517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:59.369569 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:59.369655 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:59.370152 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.370940 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.371965 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.372311 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.372434 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:59.373732 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:59.373736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:59.373749 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:59.373785 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:59.375344 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:59.375805 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:59.375807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:59.375808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:59.375884 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:59.376179 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:59.376182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:59.376183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:59.376396 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:59.376577 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:59.377000 (+ 423us) server_negotiation.cc:299] Negotiation successful
0929 16:25:59.377115 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":36}
I20260929 16:25:59.377279 1720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:59.368884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34879 (local address 127.0.98.1:41009)
0929 16:25:59.369180 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:59.369193 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:59.369386 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:59.369673 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:59.369676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:59.369894 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:59.370032 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:59.370038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.371104 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.371108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:59.371830 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:59.371840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.372440 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.372444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:59.372544 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:59.373139 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:59.373151 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:59.373581 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:59.375472 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:59.375479 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:59.375484 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:59.375702 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:59.375968 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:59.375970 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:59.375972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:59.376063 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:59.376601 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:59.376603 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:59.376695 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 16:25:59.376856 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 16:25:59.377012 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 16:25:59.397969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34879
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:25:59.515328 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:25:59.515614 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:25:59.515659 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:25:59.519863 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:25:59.519950 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:25:59.519991 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:25:59.520071 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:25:59.525741 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:25:59.526988 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:25:59.528205 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:25:59.538606 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:59.538611 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:59.538628 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:59.538837 1722 server_base.cc:1034] running on GCE node
I20260929 16:25:59.539422 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:25:59.540063 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:25:59.541280 1722 hybrid_clock.cc:648] HybridClock initialized: now 1790699159541254 us; error 37 us; skew 500 ppm
Sep 29 16:25:59 dist-test-slave-vq9v krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:25:59.544229 1722 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:25:59.545289 1722 webserver.cc:460] Webserver started at http://127.0.98.2:46459/ using document root <none> and password file <none>
I20260929 16:25:59.545847 1722 fs_manager.cc:359] Metadata directory not provided
I20260929 16:25:59.545910 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:25:59.546087 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:25:59.548183 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "a377ebe96e7c42f38ba604cbccdebfaa"
format_stamp: "Formatted at 2026-09-29 16:25:59 on dist-test-slave-vq9v"
server_key: "8f96ba81931956fa4b461c410e735bd6"
server_key_iv: "f7b25aa89b281c50014630ac307bee46"
server_key_version: "encryptionkey@0"
I20260929 16:25:59.548733 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a377ebe96e7c42f38ba604cbccdebfaa"
format_stamp: "Formatted at 2026-09-29 16:25:59 on dist-test-slave-vq9v"
server_key: "8f96ba81931956fa4b461c410e735bd6"
server_key_iv: "f7b25aa89b281c50014630ac307bee46"
server_key_version: "encryptionkey@0"
I20260929 16:25:59.552699 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:25:59.555491 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:59.556852 1722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:25:59.557005 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "a377ebe96e7c42f38ba604cbccdebfaa"
format_stamp: "Formatted at 2026-09-29 16:25:59 on dist-test-slave-vq9v"
server_key: "8f96ba81931956fa4b461c410e735bd6"
server_key_iv: "f7b25aa89b281c50014630ac307bee46"
server_key_version: "encryptionkey@0"
I20260929 16:25:59.557137 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:25:59.576778 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:25:59.580300 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:25:59.580528 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:25:59.582167 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:25:59.582243 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:59.582317 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:25:59.582362 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:25:59 dist-test-slave-vq9v krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:25:59.593173 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38323
I20260929 16:25:59.594228 1852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38323 every 8 connection(s)
I20260929 16:25:59.594419 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:25:59.595026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 1722
I20260929 16:25:59.595118 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:25:59.595413 392 external_mini_cluster.cc:1428] Setting key a5bc90abb9337cd0616c366b245971fc
I20260929 16:25:59.596439 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:59.582954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44305 (local address 127.0.98.62:34879)
0929 16:25:59.583118 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:59.583126 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:59.583995 (+ 869us) server_negotiation.cc:407] Connection header received
0929 16:25:59.585040 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:59.585044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:59.585109 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:59.585169 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:59.586717 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.587291 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.588456 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.588679 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.588796 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:59.591988 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:59.591993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:59.591997 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:59.592034 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:59.594028 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:59.594649 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:59.594652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:59.594654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:59.594736 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:59.595176 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:59.595182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:59.595184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:59.595436 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:59.595534 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:59.596212 (+ 678us) server_negotiation.cc:299] Negotiation successful
0929 16:25:59.596322 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:25:59.597524 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:59.583368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34879 (local address 127.0.98.2:44305)
0929 16:25:59.583825 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:59.583876 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:59.584787 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:59.585649 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:59.585656 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:59.586063 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:59.586564 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:59.586578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.587429 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.587432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:59.588313 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:59.588322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.588854 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.588858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:59.589065 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:59.589810 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:59.589826 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:59.591768 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:59.594172 (+ 2404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:59.594178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:59.594194 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:59.594551 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:59.594859 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:59.594863 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:59.594868 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:59.595045 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:59.595584 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:59.595590 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:59.595905 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 16:25:59.596180 (+ 275us) client_negotiation.cc:240] Negotiation successful
0929 16:25:59.596516 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":130,"threads_started":1}
I20260929 16:25:59.599025 1853 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34879
I20260929 16:25:59.599190 1853 heartbeater.cc:464] Registering TS with master...
I20260929 16:25:59.599809 1853 heartbeater.cc:511] Master 127.0.98.62:34879 requested a full tablet report, sending...
I20260929 16:25:59.600819 1509 ts_manager.cc:194] Registered new tserver with Master: a377ebe96e7c42f38ba604cbccdebfaa (127.0.98.2:38323)
I20260929 16:25:59.601560 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44305
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 16:25:59.608821 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:59.601333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53789 (local address 127.0.98.62:34879)
0929 16:25:59.601471 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:59.601476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:59.601912 (+ 436us) server_negotiation.cc:407] Connection header received
0929 16:25:59.602088 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:59.602091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:59.602133 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:59.602181 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:59.602691 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.603190 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.604026 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.604244 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.604357 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:59.605670 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:59.605674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:59.605682 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:59.605705 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:59.607108 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:59.607546 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:59.607551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:59.607553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:59.607635 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:59.607927 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:59.607931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:59.607933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:59.608152 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:59.608266 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:59.608533 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 16:25:59.608647 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 16:25:59.608834 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:59.601273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34879 (local address 127.0.98.2:53789)
0929 16:25:59.601830 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:59.601845 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:59.601982 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:59.602290 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:59.602292 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:59.602478 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:59.602574 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:59.602579 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.603316 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.603319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:59.603896 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:59.603905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.604359 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.604363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:59.604465 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:59.605065 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:59.605076 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:59.605533 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:59.607227 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:59.607231 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:59.607233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:59.607444 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:59.607728 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:59.607733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:59.607735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:59.607830 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:59.608254 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:59.608257 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:59.608360 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 16:25:59.608479 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 16:25:59.608639 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":287,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 16:25:59.629693 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34879
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:25:59.757537 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:25:59.757880 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:25:59.757929 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:25:59.762612 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:25:59.762699 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:25:59.762742 1862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:25:59.762779 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:25:59.769114 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:25:59.770287 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:25:59.771387 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:25:59.782318 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:25:59.782375 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:59.782408 1862 server_base.cc:1034] running on GCE node
W20260929 16:25:59.782331 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:25:59.783085 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:25:59.783857 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:25:59.785101 1862 hybrid_clock.cc:648] HybridClock initialized: now 1790699159785081 us; error 36 us; skew 500 ppm
Sep 29 16:25:59 dist-test-slave-vq9v krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:25:59.788271 1862 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:25:59.789331 1862 webserver.cc:460] Webserver started at http://127.0.98.3:43993/ using document root <none> and password file <none>
I20260929 16:25:59.789929 1862 fs_manager.cc:359] Metadata directory not provided
I20260929 16:25:59.790024 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:25:59.790258 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:25:59.792671 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "a6a6bd00bef7457983c31d7d81fe1855"
format_stamp: "Formatted at 2026-09-29 16:25:59 on dist-test-slave-vq9v"
server_key: "f1d4e209f2f441db18493a5271668160"
server_key_iv: "82b40db534ab2928a5e6e133f3e5b718"
server_key_version: "encryptionkey@0"
I20260929 16:25:59.793385 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a6a6bd00bef7457983c31d7d81fe1855"
format_stamp: "Formatted at 2026-09-29 16:25:59 on dist-test-slave-vq9v"
server_key: "f1d4e209f2f441db18493a5271668160"
server_key_iv: "82b40db534ab2928a5e6e133f3e5b718"
server_key_version: "encryptionkey@0"
I20260929 16:25:59.797546 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:25:59.800524 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:59.802151 1862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:25:59.802326 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "a6a6bd00bef7457983c31d7d81fe1855"
format_stamp: "Formatted at 2026-09-29 16:25:59 on dist-test-slave-vq9v"
server_key: "f1d4e209f2f441db18493a5271668160"
server_key_iv: "82b40db534ab2928a5e6e133f3e5b718"
server_key_version: "encryptionkey@0"
I20260929 16:25:59.802467 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:25:59.815902 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:25:59.819252 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:25:59.819418 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:25:59.820856 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:25:59.820922 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:25:59.820964 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:25:59.821030 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:25:59 dist-test-slave-vq9v krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:25:59.832437 1862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35015
I20260929 16:25:59.832533 1992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35015 every 8 connection(s)
I20260929 16:25:59.833763 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:25:59.836313 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:59.822067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58845 (local address 127.0.98.62:34879)
0929 16:25:59.822227 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:59.822233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:59.823122 (+ 889us) server_negotiation.cc:407] Connection header received
0929 16:25:59.824353 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:59.824357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:59.824417 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:59.824530 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:59.826298 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.827218 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.828653 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.828899 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.829022 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:59.832287 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:59.832294 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:59.832297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:59.832336 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:59.834074 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:59.834762 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:59.834766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:59.834768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:59.834875 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:59.835222 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:59.835225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:59.835226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:59.835422 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:59.835504 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:59.836062 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 16:25:59.836184 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:25:59.837077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 1862
I20260929 16:25:59.837234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:25:59.837525 392 external_mini_cluster.cc:1428] Setting key dbfec823d8de6bf1326310785b4cab4a
I20260929 16:25:59.837507 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:59.822489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34879 (local address 127.0.98.3:58845)
0929 16:25:59.822958 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:59.823012 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:59.824122 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:59.824721 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:59.824732 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:59.825256 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:59.826068 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:59.826087 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.827366 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.827370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:59.828516 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:59.828525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.829067 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.829072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:59.829350 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:59.830085 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:59.830106 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:59.832078 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:59.834229 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:59.834236 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:59.834253 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:59.834641 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:59.834984 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:59.834990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:59.834993 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:59.835137 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:59.835542 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:59.835547 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:59.835791 (+ 244us) client_negotiation.cc:769] Sending connection context
0929 16:25:59.835977 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 16:25:59.836251 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":137,"threads_started":1}
I20260929 16:25:59.839133 1993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34879
I20260929 16:25:59.839308 1993 heartbeater.cc:464] Registering TS with master...
I20260929 16:25:59.839864 1993 heartbeater.cc:511] Master 127.0.98.62:34879 requested a full tablet report, sending...
I20260929 16:25:59.841109 1506 ts_manager.cc:194] Registered new tserver with Master: a6a6bd00bef7457983c31d7d81fe1855 (127.0.98.3:35015)
I20260929 16:25:59.841894 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58845
I20260929 16:25:59.842376 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:25:59.853121 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:59.842246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38999 (local address 127.0.98.62:34879)
0929 16:25:59.844709 (+ 2463us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:59.844713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:59.844731 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:25:59.844773 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:59.844775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:59.844817 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:59.844916 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:59.845729 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.846312 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.847721 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.848003 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.848156 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:59.849608 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:59.849612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:59.849615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:59.849641 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:59.851280 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:59.851810 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:59.851814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:59.851821 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:59.851902 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:59.852215 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:59.852218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:59.852220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:59.852416 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:59.852557 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:59.852831 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 16:25:59.852949 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2366}
I20260929 16:25:59.853144 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:59.841914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34879 (local address 127.0.98.3:38999)
0929 16:25:59.842249 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:59.842265 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:59.842358 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:59.845006 (+ 2648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:59.845012 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:59.845365 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:25:59.845561 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:59.845572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.846486 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.846492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:59.847554 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:59.847570 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.848177 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.848181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:59.848321 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:59.848946 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:25:59.848958 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:25:59.849432 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:25:59.851444 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:59.851448 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:59.851450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:59.851691 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:59.852011 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:25:59.852015 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:25:59.852017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:25:59.852109 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:25:59.852543 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:25:59.852546 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:25:59.852655 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 16:25:59.852765 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 16:25:59.852937 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":128,"threads_started":1}
Sep 29 16:25:59 dist-test-slave-vq9v krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699159, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:25:59 dist-test-slave-vq9v krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699159, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:25:59.872507 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:59.862533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36174 (local address 127.0.98.62:34879)
0929 16:25:59.862689 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:59.862693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:59.862707 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:25:59.862751 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:59.862754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:59.862817 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:59.862896 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:25:59.863784 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.864374 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.865494 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.865771 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.865960 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:59.868969 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:25:59.868973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:25:59.868975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:25:59.869006 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:25:59.870521 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:59.871065 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:59.871068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:59.871069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:59.871160 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:25:59.871538 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:25:59.871541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:25:59.871543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:25:59.871764 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:25:59.871880 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:59.872260 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 16:25:59.872375 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:25:59.876844 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:25:59.880412 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:25:59.896806 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:59.890007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40661 (local address 127.0.0.1:51362)
0929 16:25:59.891237 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:59.891251 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:59.891351 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:59.891875 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:59.891882 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:59.891904 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:25:59.892100 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:59.892107 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.893618 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.893621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:59.895089 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:59.895099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.896162 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.896166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:59.896287 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:59.896326 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 16:25:59.896478 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 16:25:59.896620 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"mutex_wait_us":202,"thread_start_us":105,"threads_started":1}
I20260929 16:25:59.897300 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:59.890018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44932 (local address 127.0.98.3:35015)
0929 16:25:59.891847 (+ 1829us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:59.891857 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:59.891879 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:25:59.891963 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:59.891970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:59.892139 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:59.892274 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:25:59.892617 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.893601 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.895581 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.896336 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.896498 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:59.896647 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:59.897012 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 16:25:59.897118 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1734,"thread_start_us":104,"threads_started":1}
I20260929 16:25:59.897464 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:59.889824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35015 (local address 127.0.0.1:44932)
0929 16:25:59.890602 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:59.890657 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:59.890818 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:59.892402 (+ 1584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:59.892407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:59.892428 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:25:59.892500 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:59.892505 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.893721 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.893724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:59.895460 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:59.895467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.896498 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.896502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:59.896613 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:59.896642 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 16:25:59.896718 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 16:25:59.896792 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":141,"thread_start_us":129,"threads_started":1}
I20260929 16:25:59.896839 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:59.890205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38323 (local address 127.0.0.1:51644)
0929 16:25:59.891530 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0929 16:25:59.891543 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:25:59.891616 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:25:59.892197 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:25:59.892203 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:25:59.892225 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:25:59.892327 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:59.892334 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.893826 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.893830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:59.895366 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:25:59.895376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.896473 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.896477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:25:59.896557 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:59.896611 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 16:25:59.896675 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 16:25:59.896737 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1249,"mutex_wait_us":126,"thread_start_us":99,"threads_started":1}
I20260929 16:25:59.897585 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:59.891236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51644 (local address 127.0.98.2:38323)
0929 16:25:59.891586 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:59.891603 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:59.891626 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:25:59.891707 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:59.891712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:59.891905 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:59.892052 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:25:59.892456 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.893658 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.895517 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.896149 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.896255 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:59.896413 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:59.896907 (+ 494us) server_negotiation.cc:299] Negotiation successful
0929 16:25:59.897260 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":139,"threads_started":1}
I20260929 16:25:59.898515 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:25:59.890715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51362 (local address 127.0.98.1:40661)
0929 16:25:59.890971 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 16:25:59.890979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:25:59.891357 (+ 378us) server_negotiation.cc:407] Connection header received
0929 16:25:59.891463 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:25:59.891468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:25:59.891716 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:25:59.891893 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:25:59.892266 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.893489 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.895232 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:25:59.896004 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:25:59.896119 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:25:59.896271 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 16:25:59.897012 (+ 741us) server_negotiation.cc:299] Negotiation successful
0929 16:25:59.897120 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":94,"threads_started":1}
I20260929 16:25:59.900483 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 82ae56827bb24c9aaff6d5d9aad003fb (DEFAULT_TABLE table=test-table [id=b95238a68bb447dd9a22c8df2f6aa8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:25:59.900501 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 82ae56827bb24c9aaff6d5d9aad003fb (DEFAULT_TABLE table=test-table [id=b95238a68bb447dd9a22c8df2f6aa8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:25:59.901032 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 82ae56827bb24c9aaff6d5d9aad003fb (DEFAULT_TABLE table=test-table [id=b95238a68bb447dd9a22c8df2f6aa8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:25:59.901629 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82ae56827bb24c9aaff6d5d9aad003fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:25:59.901813 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82ae56827bb24c9aaff6d5d9aad003fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:25:59.901952 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82ae56827bb24c9aaff6d5d9aad003fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:25:59.908561 2016 tablet_bootstrap.cc:492] T 82ae56827bb24c9aaff6d5d9aad003fb P a6a6bd00bef7457983c31d7d81fe1855: Bootstrap starting.
I20260929 16:25:59.910089 2017 tablet_bootstrap.cc:492] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa: Bootstrap starting.
I20260929 16:25:59.910998 2016 tablet_bootstrap.cc:654] T 82ae56827bb24c9aaff6d5d9aad003fb P a6a6bd00bef7457983c31d7d81fe1855: Neither blocks nor log segments found. Creating new log.
I20260929 16:25:59.911780 2016 log.cc:824] T 82ae56827bb24c9aaff6d5d9aad003fb P a6a6bd00bef7457983c31d7d81fe1855: Log is configured to *not* fsync() on all Append() calls
I20260929 16:25:59.912038 2017 tablet_bootstrap.cc:654] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa: Neither blocks nor log segments found. Creating new log.
I20260929 16:25:59.912856 2017 log.cc:824] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa: Log is configured to *not* fsync() on all Append() calls
I20260929 16:25:59.913012 2018 tablet_bootstrap.cc:492] T 82ae56827bb24c9aaff6d5d9aad003fb P 4f5fa3284cff4be68ddf362d78991ce0: Bootstrap starting.
I20260929 16:25:59.914140 2016 tablet_bootstrap.cc:492] T 82ae56827bb24c9aaff6d5d9aad003fb P a6a6bd00bef7457983c31d7d81fe1855: No bootstrap required, opened a new log
I20260929 16:25:59.914347 2016 ts_tablet_manager.cc:1397] T 82ae56827bb24c9aaff6d5d9aad003fb P a6a6bd00bef7457983c31d7d81fe1855: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:25:59.914891 2017 tablet_bootstrap.cc:492] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa: No bootstrap required, opened a new log
I20260929 16:25:59.915079 2017 ts_tablet_manager.cc:1397] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260929 16:25:59.915074 2018 tablet_bootstrap.cc:654] T 82ae56827bb24c9aaff6d5d9aad003fb P 4f5fa3284cff4be68ddf362d78991ce0: Neither blocks nor log segments found. Creating new log.
I20260929 16:25:59.915746 2018 log.cc:824] T 82ae56827bb24c9aaff6d5d9aad003fb P 4f5fa3284cff4be68ddf362d78991ce0: Log is configured to *not* fsync() on all Append() calls
I20260929 16:25:59.917479 2018 tablet_bootstrap.cc:492] T 82ae56827bb24c9aaff6d5d9aad003fb P 4f5fa3284cff4be68ddf362d78991ce0: No bootstrap required, opened a new log
I20260929 16:25:59.917668 2018 ts_tablet_manager.cc:1397] T 82ae56827bb24c9aaff6d5d9aad003fb P 4f5fa3284cff4be68ddf362d78991ce0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:25:59.918390 2016 raft_consensus.cc:348] T 82ae56827bb24c9aaff6d5d9aad003fb P a6a6bd00bef7457983c31d7d81fe1855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } }
I20260929 16:25:59.918682 2016 raft_consensus.cc:374] T 82ae56827bb24c9aaff6d5d9aad003fb P a6a6bd00bef7457983c31d7d81fe1855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:25:59.918746 2016 raft_consensus.cc:729] T 82ae56827bb24c9aaff6d5d9aad003fb P a6a6bd00bef7457983c31d7d81fe1855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a6bd00bef7457983c31d7d81fe1855, State: Initialized, Role: FOLLOWER
I20260929 16:25:59.918888 2017 raft_consensus.cc:348] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } }
I20260929 16:25:59.919137 2017 raft_consensus.cc:374] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:25:59.919188 2017 raft_consensus.cc:729] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a377ebe96e7c42f38ba604cbccdebfaa, State: Initialized, Role: FOLLOWER
I20260929 16:25:59.919201 2016 consensus_queue.cc:260] T 82ae56827bb24c9aaff6d5d9aad003fb P a6a6bd00bef7457983c31d7d81fe1855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } }
I20260929 16:25:59.919639 2017 consensus_queue.cc:260] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } }
I20260929 16:25:59.920025 1993 heartbeater.cc:503] Master 127.0.98.62:34879 was elected leader, sending a full tablet report...
I20260929 16:25:59.920324 2016 ts_tablet_manager.cc:1428] T 82ae56827bb24c9aaff6d5d9aad003fb P a6a6bd00bef7457983c31d7d81fe1855: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260929 16:25:59.920646 2017 ts_tablet_manager.cc:1428] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260929 16:25:59.920959 1853 heartbeater.cc:503] Master 127.0.98.62:34879 was elected leader, sending a full tablet report...
I20260929 16:25:59.921974 2018 raft_consensus.cc:348] T 82ae56827bb24c9aaff6d5d9aad003fb P 4f5fa3284cff4be68ddf362d78991ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } }
I20260929 16:25:59.922238 2018 raft_consensus.cc:374] T 82ae56827bb24c9aaff6d5d9aad003fb P 4f5fa3284cff4be68ddf362d78991ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:25:59.922310 2018 raft_consensus.cc:729] T 82ae56827bb24c9aaff6d5d9aad003fb P 4f5fa3284cff4be68ddf362d78991ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5fa3284cff4be68ddf362d78991ce0, State: Initialized, Role: FOLLOWER
I20260929 16:25:59.922793 2018 consensus_queue.cc:260] T 82ae56827bb24c9aaff6d5d9aad003fb P 4f5fa3284cff4be68ddf362d78991ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } }
I20260929 16:25:59.923837 2018 ts_tablet_manager.cc:1428] T 82ae56827bb24c9aaff6d5d9aad003fb P 4f5fa3284cff4be68ddf362d78991ce0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:25:59.924191 1713 heartbeater.cc:503] Master 127.0.98.62:34879 was elected leader, sending a full tablet report...
W20260929 16:26:00.085068 1994 tablet.cc:2369] T 82ae56827bb24c9aaff6d5d9aad003fb P a6a6bd00bef7457983c31d7d81fe1855: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:26:00.095772 1854 tablet.cc:2369] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:26:00.113359 1714 tablet.cc:2369] T 82ae56827bb24c9aaff6d5d9aad003fb P 4f5fa3284cff4be68ddf362d78991ce0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:26:00.190589 2023 raft_consensus.cc:482] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:26:00.190912 2023 raft_consensus.cc:504] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } }
I20260929 16:26:00.192341 2023 leader_election.cc:288] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6a6bd00bef7457983c31d7d81fe1855 (127.0.98.3:35015), 4f5fa3284cff4be68ddf362d78991ce0 (127.0.98.1:40661)
I20260929 16:26:00.198872 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.192909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40239 (local address 127.0.98.1:40661)
0929 16:26:00.193121 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:00.193126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:00.193519 (+ 393us) server_negotiation.cc:407] Connection header received
0929 16:26:00.193579 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:00.193582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:00.193639 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:00.193727 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:00.194245 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.195166 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.197321 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.197969 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.198103 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.198178 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:00.198611 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 16:26:00.198701 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260929 16:26:00.198843 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.192887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37957 (local address 127.0.98.3:35015)
0929 16:26:00.193065 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:00.193070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:00.193173 (+ 103us) server_negotiation.cc:407] Connection header received
0929 16:26:00.193581 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:00.193584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:00.193639 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:00.193727 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:00.194245 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.195202 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.197280 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.197984 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.198103 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.198178 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:00.198611 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 16:26:00.198701 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 16:26:00.198890 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.192753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40661 (local address 127.0.98.2:40239)
0929 16:26:00.193306 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:00.193320 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:00.193403 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:00.193754 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:00.193758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:00.193803 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:00.194071 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:00.194077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.195383 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.195386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:00.197030 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:00.197047 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.198185 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.198189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:00.198397 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.198434 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:26:00.198541 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 16:26:00.198664 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":483,"spinlock_wait_cycles":8192,"thread_start_us":99,"threads_started":1}
I20260929 16:26:00.198890 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.192755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35015 (local address 127.0.98.2:37957)
0929 16:26:00.193064 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:00.193081 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:00.193250 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:00.193754 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:00.193758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:00.193801 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:00.194065 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:00.194073 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.195371 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.195376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:00.197030 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:00.197047 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.198185 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.198189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:00.198371 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.198426 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 16:26:00.198530 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 16:26:00.198664 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":157,"spinlock_wait_cycles":30720,"thread_start_us":93,"threads_started":1}
I20260929 16:26:00.199765 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae56827bb24c9aaff6d5d9aad003fb" candidate_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5fa3284cff4be68ddf362d78991ce0" is_pre_election: true
I20260929 16:26:00.199950 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae56827bb24c9aaff6d5d9aad003fb" candidate_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a6bd00bef7457983c31d7d81fe1855" is_pre_election: true
I20260929 16:26:00.200153 1667 raft_consensus.cc:2454] T 82ae56827bb24c9aaff6d5d9aad003fb P 4f5fa3284cff4be68ddf362d78991ce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a377ebe96e7c42f38ba604cbccdebfaa in term 0.
I20260929 16:26:00.200253 1947 raft_consensus.cc:2454] T 82ae56827bb24c9aaff6d5d9aad003fb P a6a6bd00bef7457983c31d7d81fe1855 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a377ebe96e7c42f38ba604cbccdebfaa in term 0.
I20260929 16:26:00.200752 1739 leader_election.cc:302] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4f5fa3284cff4be68ddf362d78991ce0, a377ebe96e7c42f38ba604cbccdebfaa, a6a6bd00bef7457983c31d7d81fe1855; no voters:
I20260929 16:26:00.201090 2023 raft_consensus.cc:2781] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:26:00.201187 2023 raft_consensus.cc:482] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:26:00.201292 2023 raft_consensus.cc:3037] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:00.202493 2023 raft_consensus.cc:504] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } }
I20260929 16:26:00.202967 2023 leader_election.cc:288] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa [CANDIDATE]: Term 1 election: Requested vote from peers a6a6bd00bef7457983c31d7d81fe1855 (127.0.98.3:35015), 4f5fa3284cff4be68ddf362d78991ce0 (127.0.98.1:40661)
I20260929 16:26:00.203408 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae56827bb24c9aaff6d5d9aad003fb" candidate_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a6bd00bef7457983c31d7d81fe1855"
I20260929 16:26:00.203562 1947 raft_consensus.cc:3037] T 82ae56827bb24c9aaff6d5d9aad003fb P a6a6bd00bef7457983c31d7d81fe1855 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:00.205893 1947 raft_consensus.cc:2454] T 82ae56827bb24c9aaff6d5d9aad003fb P a6a6bd00bef7457983c31d7d81fe1855 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a377ebe96e7c42f38ba604cbccdebfaa in term 1.
I20260929 16:26:00.206604 1738 leader_election.cc:302] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a377ebe96e7c42f38ba604cbccdebfaa, a6a6bd00bef7457983c31d7d81fe1855; no voters:
I20260929 16:26:00.207046 2023 raft_consensus.cc:2781] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:00.207307 2023 raft_consensus.cc:686] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa [term 1 LEADER]: Becoming Leader. State: Replica: a377ebe96e7c42f38ba604cbccdebfaa, State: Running, Role: LEADER
I20260929 16:26:00.207629 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae56827bb24c9aaff6d5d9aad003fb" candidate_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5fa3284cff4be68ddf362d78991ce0"
I20260929 16:26:00.207809 1667 raft_consensus.cc:3037] T 82ae56827bb24c9aaff6d5d9aad003fb P 4f5fa3284cff4be68ddf362d78991ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:00.207736 2023 consensus_queue.cc:237] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } }
I20260929 16:26:00.209349 1667 raft_consensus.cc:2454] T 82ae56827bb24c9aaff6d5d9aad003fb P 4f5fa3284cff4be68ddf362d78991ce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a377ebe96e7c42f38ba604cbccdebfaa in term 1.
I20260929 16:26:00.213178 1506 catalog_manager.cc:5495] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa reported cstate change: term changed from 0 to 1, leader changed from <none> to a377ebe96e7c42f38ba604cbccdebfaa (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } health_report { overall_health: HEALTHY } } }
I20260929 16:26:00.272898 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.268454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51660 (local address 127.0.98.2:38323)
0929 16:26:00.268616 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:00.268620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:00.268637 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:26:00.268883 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:00.268886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:00.268956 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:00.269052 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:00.269389 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.269930 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.270898 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.271124 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.271224 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.271731 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:00.272247 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:00.272350 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:00.272569 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 16:26:00.272693 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:26:00.280081 1947 raft_consensus.cc:1261] T 82ae56827bb24c9aaff6d5d9aad003fb P a6a6bd00bef7457983c31d7d81fe1855 [term 1 FOLLOWER]: Refusing update from remote peer a377ebe96e7c42f38ba604cbccdebfaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:00.280157 1667 raft_consensus.cc:1261] T 82ae56827bb24c9aaff6d5d9aad003fb P 4f5fa3284cff4be68ddf362d78991ce0 [term 1 FOLLOWER]: Refusing update from remote peer a377ebe96e7c42f38ba604cbccdebfaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:00.280905 2027 consensus_queue.cc:1035] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:00.281155 2023 consensus_queue.cc:1035] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:00.291404 2031 mvcc.cc:205] Tried to move back new op lower bound from 7334703760498225152 to 7334703760219136000. Current Snapshot: MvccSnapshot[applied={T|T < 7334703760498225152}]
I20260929 16:26:00.291392 2032 mvcc.cc:205] Tried to move back new op lower bound from 7334703760498225152 to 7334703760219136000. Current Snapshot: MvccSnapshot[applied={T|T < 7334703760498225152}]
I20260929 16:26:00.305130 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:00.305469 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:00.308955 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7: Sending DeleteTablet for 3 replicas of tablet 82ae56827bb24c9aaff6d5d9aad003fb
I20260929 16:26:00.310148 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 82ae56827bb24c9aaff6d5d9aad003fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51362
I20260929 16:26:00.310158 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 82ae56827bb24c9aaff6d5d9aad003fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44932
I20260929 16:26:00.310755 2040 tablet_replica.cc:330] T 82ae56827bb24c9aaff6d5d9aad003fb P a6a6bd00bef7457983c31d7d81fe1855: stopping tablet replica
I20260929 16:26:00.310791 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 82ae56827bb24c9aaff6d5d9aad003fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51644
I20260929 16:26:00.310874 2041 tablet_replica.cc:330] T 82ae56827bb24c9aaff6d5d9aad003fb P 4f5fa3284cff4be68ddf362d78991ce0: stopping tablet replica
I20260929 16:26:00.311240 2041 raft_consensus.cc:2229] T 82ae56827bb24c9aaff6d5d9aad003fb P 4f5fa3284cff4be68ddf362d78991ce0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:00.311241 2040 raft_consensus.cc:2229] T 82ae56827bb24c9aaff6d5d9aad003fb P a6a6bd00bef7457983c31d7d81fe1855 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:00.311685 2041 raft_consensus.cc:2258] T 82ae56827bb24c9aaff6d5d9aad003fb P 4f5fa3284cff4be68ddf362d78991ce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:00.311686 2040 raft_consensus.cc:2258] T 82ae56827bb24c9aaff6d5d9aad003fb P a6a6bd00bef7457983c31d7d81fe1855 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:00.312536 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:26:00.313151 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:26:00.313932 2040 ts_tablet_manager.cc:1905] T 82ae56827bb24c9aaff6d5d9aad003fb P a6a6bd00bef7457983c31d7d81fe1855: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:00.315428 2041 ts_tablet_manager.cc:1905] T 82ae56827bb24c9aaff6d5d9aad003fb P 4f5fa3284cff4be68ddf362d78991ce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:00.317025 2040 ts_tablet_manager.cc:1918] T 82ae56827bb24c9aaff6d5d9aad003fb P a6a6bd00bef7457983c31d7d81fe1855: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:26:00.317149 2040 log.cc:1196] T 82ae56827bb24c9aaff6d5d9aad003fb P a6a6bd00bef7457983c31d7d81fe1855: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/82ae56827bb24c9aaff6d5d9aad003fb
I20260929 16:26:00.317566 2040 ts_tablet_manager.cc:1939] T 82ae56827bb24c9aaff6d5d9aad003fb P a6a6bd00bef7457983c31d7d81fe1855: Deleting consensus metadata
I20260929 16:26:00.318542 2041 ts_tablet_manager.cc:1918] T 82ae56827bb24c9aaff6d5d9aad003fb P 4f5fa3284cff4be68ddf362d78991ce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:26:00.318660 2041 log.cc:1196] T 82ae56827bb24c9aaff6d5d9aad003fb P 4f5fa3284cff4be68ddf362d78991ce0: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-0/wal/wals/82ae56827bb24c9aaff6d5d9aad003fb
I20260929 16:26:00.319007 2041 ts_tablet_manager.cc:1939] T 82ae56827bb24c9aaff6d5d9aad003fb P 4f5fa3284cff4be68ddf362d78991ce0: Deleting consensus metadata
I20260929 16:26:00.319415 2042 tablet_replica.cc:330] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa: stopping tablet replica
I20260929 16:26:00.319727 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 8f3d771af60f4e6792e269ea945d2f1a (DEFAULT_TABLE table=test-table [id=d214e13ac6b740faa49f8d7f9d0f2c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:00.319834 2042 raft_consensus.cc:2229] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:26:00.319989 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3d771af60f4e6792e269ea945d2f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:00.320334 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 8f3d771af60f4e6792e269ea945d2f1a (DEFAULT_TABLE table=test-table [id=d214e13ac6b740faa49f8d7f9d0f2c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:00.320636 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3d771af60f4e6792e269ea945d2f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:00.321796 1495 catalog_manager.cc:4841] TS 4f5fa3284cff4be68ddf362d78991ce0 (127.0.98.1:40661): tablet 82ae56827bb24c9aaff6d5d9aad003fb (table test-table [id=b95238a68bb447dd9a22c8df2f6aa8eb]) successfully deleted
I20260929 16:26:00.322032 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 8f3d771af60f4e6792e269ea945d2f1a (DEFAULT_TABLE table=test-table [id=d214e13ac6b740faa49f8d7f9d0f2c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:00.322330 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f3d771af60f4e6792e269ea945d2f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:00.323400 2018 tablet_bootstrap.cc:492] T 8f3d771af60f4e6792e269ea945d2f1a P 4f5fa3284cff4be68ddf362d78991ce0: Bootstrap starting.
I20260929 16:26:00.324630 2018 tablet_bootstrap.cc:654] T 8f3d771af60f4e6792e269ea945d2f1a P 4f5fa3284cff4be68ddf362d78991ce0: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:00.325026 2016 tablet_bootstrap.cc:492] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855: Bootstrap starting.
I20260929 16:26:00.325758 2017 tablet_bootstrap.cc:492] T 8f3d771af60f4e6792e269ea945d2f1a P a377ebe96e7c42f38ba604cbccdebfaa: Bootstrap starting.
I20260929 16:26:00.326086 2016 tablet_bootstrap.cc:654] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:00.320338 2042 raft_consensus.cc:2258] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:00.326892 2017 tablet_bootstrap.cc:654] T 8f3d771af60f4e6792e269ea945d2f1a P a377ebe96e7c42f38ba604cbccdebfaa: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:00.327711 1494 catalog_manager.cc:4841] TS a6a6bd00bef7457983c31d7d81fe1855 (127.0.98.3:35015): tablet 82ae56827bb24c9aaff6d5d9aad003fb (table test-table [id=b95238a68bb447dd9a22c8df2f6aa8eb]) successfully deleted
I20260929 16:26:00.328069 2042 ts_tablet_manager.cc:1905] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:00.328819 2016 tablet_bootstrap.cc:492] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855: No bootstrap required, opened a new log
I20260929 16:26:00.328910 2016 ts_tablet_manager.cc:1397] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 16:26:00.329098 2017 tablet_bootstrap.cc:492] T 8f3d771af60f4e6792e269ea945d2f1a P a377ebe96e7c42f38ba604cbccdebfaa: No bootstrap required, opened a new log
I20260929 16:26:00.329185 2017 ts_tablet_manager.cc:1397] T 8f3d771af60f4e6792e269ea945d2f1a P a377ebe96e7c42f38ba604cbccdebfaa: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 16:26:00.329422 2016 raft_consensus.cc:348] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } }
I20260929 16:26:00.329548 2016 raft_consensus.cc:374] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:00.329579 2016 raft_consensus.cc:729] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a6bd00bef7457983c31d7d81fe1855, State: Initialized, Role: FOLLOWER
I20260929 16:26:00.329725 2016 consensus_queue.cc:260] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } }
I20260929 16:26:00.329767 2017 raft_consensus.cc:348] T 8f3d771af60f4e6792e269ea945d2f1a P a377ebe96e7c42f38ba604cbccdebfaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } }
I20260929 16:26:00.329909 2017 raft_consensus.cc:374] T 8f3d771af60f4e6792e269ea945d2f1a P a377ebe96e7c42f38ba604cbccdebfaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:00.329953 2017 raft_consensus.cc:729] T 8f3d771af60f4e6792e269ea945d2f1a P a377ebe96e7c42f38ba604cbccdebfaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a377ebe96e7c42f38ba604cbccdebfaa, State: Initialized, Role: FOLLOWER
I20260929 16:26:00.329999 2016 ts_tablet_manager.cc:1428] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:26:00.330098 2017 consensus_queue.cc:260] T 8f3d771af60f4e6792e269ea945d2f1a P a377ebe96e7c42f38ba604cbccdebfaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } }
I20260929 16:26:00.330395 2017 ts_tablet_manager.cc:1428] T 8f3d771af60f4e6792e269ea945d2f1a P a377ebe96e7c42f38ba604cbccdebfaa: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 16:26:00.331403 2042 ts_tablet_manager.cc:1918] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:26:00.331542 2042 log.cc:1196] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/82ae56827bb24c9aaff6d5d9aad003fb
I20260929 16:26:00.331907 2042 ts_tablet_manager.cc:1939] T 82ae56827bb24c9aaff6d5d9aad003fb P a377ebe96e7c42f38ba604cbccdebfaa: Deleting consensus metadata
I20260929 16:26:00.332242 2018 tablet_bootstrap.cc:492] T 8f3d771af60f4e6792e269ea945d2f1a P 4f5fa3284cff4be68ddf362d78991ce0: No bootstrap required, opened a new log
I20260929 16:26:00.332306 2022 raft_consensus.cc:482] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:26:00.332358 2018 ts_tablet_manager.cc:1397] T 8f3d771af60f4e6792e269ea945d2f1a P 4f5fa3284cff4be68ddf362d78991ce0: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260929 16:26:00.332408 2022 raft_consensus.cc:504] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } }
I20260929 16:26:00.333297 2018 raft_consensus.cc:348] T 8f3d771af60f4e6792e269ea945d2f1a P 4f5fa3284cff4be68ddf362d78991ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } }
I20260929 16:26:00.333444 2018 raft_consensus.cc:374] T 8f3d771af60f4e6792e269ea945d2f1a P 4f5fa3284cff4be68ddf362d78991ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:00.333482 2018 raft_consensus.cc:729] T 8f3d771af60f4e6792e269ea945d2f1a P 4f5fa3284cff4be68ddf362d78991ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5fa3284cff4be68ddf362d78991ce0, State: Initialized, Role: FOLLOWER
I20260929 16:26:00.333444 1497 catalog_manager.cc:4841] TS a377ebe96e7c42f38ba604cbccdebfaa (127.0.98.2:38323): tablet 82ae56827bb24c9aaff6d5d9aad003fb (table test-table [id=b95238a68bb447dd9a22c8df2f6aa8eb]) successfully deleted
I20260929 16:26:00.333616 2018 consensus_queue.cc:260] T 8f3d771af60f4e6792e269ea945d2f1a P 4f5fa3284cff4be68ddf362d78991ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } }
I20260929 16:26:00.333662 2022 leader_election.cc:288] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f5fa3284cff4be68ddf362d78991ce0 (127.0.98.1:40661), a377ebe96e7c42f38ba604cbccdebfaa (127.0.98.2:38323)
I20260929 16:26:00.333928 2018 ts_tablet_manager.cc:1428] T 8f3d771af60f4e6792e269ea945d2f1a P 4f5fa3284cff4be68ddf362d78991ce0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 16:26:00.339191 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.334115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51171 (local address 127.0.98.1:40661)
0929 16:26:00.334246 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:00.334253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:00.334691 (+ 438us) server_negotiation.cc:407] Connection header received
0929 16:26:00.334857 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:00.334862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:00.334913 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:00.334962 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:00.335383 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.336103 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.337689 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.338508 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.338595 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.338688 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:00.338985 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 16:26:00.339065 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 16:26:00.339208 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.334037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40661 (local address 127.0.98.3:51171)
0929 16:26:00.334592 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:00.334608 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:00.334738 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:00.335114 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:00.335118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:00.335136 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:00.335263 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:00.335270 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.336239 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.336242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:00.337532 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:00.337542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.338733 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.338736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:00.338840 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.338860 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 16:26:00.338965 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 16:26:00.339045 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483}
I20260929 16:26:00.339941 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.334948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44819 (local address 127.0.98.2:38323)
0929 16:26:00.335077 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:00.335083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:00.335097 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:26:00.335140 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:00.335142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:00.335182 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:00.335248 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:00.336162 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.336910 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.338539 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.339178 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.339299 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.339370 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:00.339732 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 16:26:00.339817 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:26:00.340070 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3d771af60f4e6792e269ea945d2f1a" candidate_uuid: "a6a6bd00bef7457983c31d7d81fe1855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5fa3284cff4be68ddf362d78991ce0" is_pre_election: true
I20260929 16:26:00.340277 1667 raft_consensus.cc:2454] T 8f3d771af60f4e6792e269ea945d2f1a P 4f5fa3284cff4be68ddf362d78991ce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6a6bd00bef7457983c31d7d81fe1855 in term 0.
I20260929 16:26:00.340374 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.334321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38323 (local address 127.0.98.3:44819)
0929 16:26:00.334626 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:00.334641 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:00.334721 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:00.335533 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:00.335535 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:00.335553 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:00.335668 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:00.335674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.337063 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.337067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:00.338321 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:00.338334 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.339436 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.339441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:00.339580 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.339606 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 16:26:00.339691 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 16:26:00.339787 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":120,"threads_started":1}
I20260929 16:26:00.341006 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3d771af60f4e6792e269ea945d2f1a" candidate_uuid: "a6a6bd00bef7457983c31d7d81fe1855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" is_pre_election: true
I20260929 16:26:00.341107 1879 leader_election.cc:302] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f5fa3284cff4be68ddf362d78991ce0, a6a6bd00bef7457983c31d7d81fe1855; no voters:
I20260929 16:26:00.341312 1807 raft_consensus.cc:2454] T 8f3d771af60f4e6792e269ea945d2f1a P a377ebe96e7c42f38ba604cbccdebfaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6a6bd00bef7457983c31d7d81fe1855 in term 0.
I20260929 16:26:00.341785 2022 raft_consensus.cc:2781] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:26:00.341866 2022 raft_consensus.cc:482] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:26:00.341920 2022 raft_consensus.cc:3037] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:00.342937 2022 raft_consensus.cc:504] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } }
I20260929 16:26:00.343353 2022 leader_election.cc:288] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855 [CANDIDATE]: Term 1 election: Requested vote from peers 4f5fa3284cff4be68ddf362d78991ce0 (127.0.98.1:40661), a377ebe96e7c42f38ba604cbccdebfaa (127.0.98.2:38323)
I20260929 16:26:00.343726 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3d771af60f4e6792e269ea945d2f1a" candidate_uuid: "a6a6bd00bef7457983c31d7d81fe1855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5fa3284cff4be68ddf362d78991ce0"
I20260929 16:26:00.343881 1667 raft_consensus.cc:3037] T 8f3d771af60f4e6792e269ea945d2f1a P 4f5fa3284cff4be68ddf362d78991ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:00.343880 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f3d771af60f4e6792e269ea945d2f1a" candidate_uuid: "a6a6bd00bef7457983c31d7d81fe1855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a377ebe96e7c42f38ba604cbccdebfaa"
I20260929 16:26:00.344004 1807 raft_consensus.cc:3037] T 8f3d771af60f4e6792e269ea945d2f1a P a377ebe96e7c42f38ba604cbccdebfaa [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:00.345160 1667 raft_consensus.cc:2454] T 8f3d771af60f4e6792e269ea945d2f1a P 4f5fa3284cff4be68ddf362d78991ce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a6bd00bef7457983c31d7d81fe1855 in term 1.
I20260929 16:26:00.345160 1807 raft_consensus.cc:2454] T 8f3d771af60f4e6792e269ea945d2f1a P a377ebe96e7c42f38ba604cbccdebfaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6a6bd00bef7457983c31d7d81fe1855 in term 1.
I20260929 16:26:00.345541 1879 leader_election.cc:302] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f5fa3284cff4be68ddf362d78991ce0, a6a6bd00bef7457983c31d7d81fe1855; no voters:
I20260929 16:26:00.345788 2022 raft_consensus.cc:2781] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:00.346184 2022 raft_consensus.cc:686] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855 [term 1 LEADER]: Becoming Leader. State: Replica: a6a6bd00bef7457983c31d7d81fe1855, State: Running, Role: LEADER
I20260929 16:26:00.346737 2022 consensus_queue.cc:237] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } }
I20260929 16:26:00.349587 1506 catalog_manager.cc:5495] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6a6bd00bef7457983c31d7d81fe1855 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a6a6bd00bef7457983c31d7d81fe1855" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } health_report { overall_health: HEALTHY } } }
Sep 29 16:26:00 dist-test-slave-vq9v krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:00.395185 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.383211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34879 (local address 127.0.0.1:36180)
0929 16:26:00.383590 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:00.383614 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:00.383735 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:00.384112 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:00.384114 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:00.384601 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:00.384710 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:00.384717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.385946 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.385949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:00.386561 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:00.386571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.387250 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.387254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:00.387378 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.388168 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:00.388182 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:00.390078 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:00.392334 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:00.392351 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:00.392360 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:00.392753 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:00.393481 (+ 728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:00.393484 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:00.393485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:00.393627 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:00.394167 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:00.394174 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:00.394303 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 16:26:00.394417 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 16:26:00.395031 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":148,"threads_started":1}
I20260929 16:26:00.395391 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.383300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36180 (local address 127.0.98.62:34879)
0929 16:26:00.383611 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:00.383620 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:00.383704 (+ 84us) server_negotiation.cc:407] Connection header received
0929 16:26:00.383838 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:00.383842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:00.383900 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:00.384035 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:00.384868 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.385786 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.386691 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.387071 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.387200 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.390289 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:00.390293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:00.390297 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:00.390326 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:00.392184 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:00.392877 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:00.392880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:00.392882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:00.393013 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:00.393743 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:00.393747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:00.393749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:00.394021 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:00.394133 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:00.394677 (+ 544us) server_negotiation.cc:299] Negotiation successful
0929 16:26:00.395150 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":289,"server-negotiator.queue_time_us":235,"thread_start_us":120,"threads_started":1}
I20260929 16:26:00.400944 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36180:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 16:26:00.401963 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:26:00.408435 1647 tablet_service.cc:1467] Processing CreateTablet for tablet dd640a6ae5a94eea9467032fbee8a89b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a0b237426dbf4dbcb9c1c54bd8b7fffd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:26:00.408456 1787 tablet_service.cc:1467] Processing CreateTablet for tablet dd640a6ae5a94eea9467032fbee8a89b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a0b237426dbf4dbcb9c1c54bd8b7fffd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:26:00.408556 1927 tablet_service.cc:1467] Processing CreateTablet for tablet dd640a6ae5a94eea9467032fbee8a89b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a0b237426dbf4dbcb9c1c54bd8b7fffd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:26:00.408901 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd640a6ae5a94eea9467032fbee8a89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:00.408902 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd640a6ae5a94eea9467032fbee8a89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:00.408980 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd640a6ae5a94eea9467032fbee8a89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:00.412335 2016 tablet_bootstrap.cc:492] T dd640a6ae5a94eea9467032fbee8a89b P a6a6bd00bef7457983c31d7d81fe1855: Bootstrap starting.
I20260929 16:26:00.413735 2016 tablet_bootstrap.cc:654] T dd640a6ae5a94eea9467032fbee8a89b P a6a6bd00bef7457983c31d7d81fe1855: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:00.414510 2018 tablet_bootstrap.cc:492] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0: Bootstrap starting.
I20260929 16:26:00.415202 2016 tablet_bootstrap.cc:492] T dd640a6ae5a94eea9467032fbee8a89b P a6a6bd00bef7457983c31d7d81fe1855: No bootstrap required, opened a new log
I20260929 16:26:00.415334 2016 ts_tablet_manager.cc:1397] T dd640a6ae5a94eea9467032fbee8a89b P a6a6bd00bef7457983c31d7d81fe1855: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:26:00.415519 2018 tablet_bootstrap.cc:654] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:00.416229 2017 tablet_bootstrap.cc:492] T dd640a6ae5a94eea9467032fbee8a89b P a377ebe96e7c42f38ba604cbccdebfaa: Bootstrap starting.
I20260929 16:26:00.416160 2016 raft_consensus.cc:348] T dd640a6ae5a94eea9467032fbee8a89b P a6a6bd00bef7457983c31d7d81fe1855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } }
I20260929 16:26:00.416312 2016 raft_consensus.cc:374] T dd640a6ae5a94eea9467032fbee8a89b P a6a6bd00bef7457983c31d7d81fe1855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:00.416353 2016 raft_consensus.cc:729] T dd640a6ae5a94eea9467032fbee8a89b P a6a6bd00bef7457983c31d7d81fe1855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a6bd00bef7457983c31d7d81fe1855, State: Initialized, Role: FOLLOWER
I20260929 16:26:00.416531 2016 consensus_queue.cc:260] T dd640a6ae5a94eea9467032fbee8a89b P a6a6bd00bef7457983c31d7d81fe1855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } }
I20260929 16:26:00.416841 2022 tablet_replica.cc:439] T dd640a6ae5a94eea9467032fbee8a89b P a6a6bd00bef7457983c31d7d81fe1855: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } }
I20260929 16:26:00.416980 2022 tablet_replica.cc:442] T dd640a6ae5a94eea9467032fbee8a89b P a6a6bd00bef7457983c31d7d81fe1855: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:26:00.417205 2016 ts_tablet_manager.cc:1428] T dd640a6ae5a94eea9467032fbee8a89b P a6a6bd00bef7457983c31d7d81fe1855: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:26:00.417255 2017 tablet_bootstrap.cc:654] T dd640a6ae5a94eea9467032fbee8a89b P a377ebe96e7c42f38ba604cbccdebfaa: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:00.418893 2017 tablet_bootstrap.cc:492] T dd640a6ae5a94eea9467032fbee8a89b P a377ebe96e7c42f38ba604cbccdebfaa: No bootstrap required, opened a new log
I20260929 16:26:00.418993 2017 ts_tablet_manager.cc:1397] T dd640a6ae5a94eea9467032fbee8a89b P a377ebe96e7c42f38ba604cbccdebfaa: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 16:26:00.419451 2017 raft_consensus.cc:348] T dd640a6ae5a94eea9467032fbee8a89b P a377ebe96e7c42f38ba604cbccdebfaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } }
I20260929 16:26:00.419576 2017 raft_consensus.cc:374] T dd640a6ae5a94eea9467032fbee8a89b P a377ebe96e7c42f38ba604cbccdebfaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:00.419615 2017 raft_consensus.cc:729] T dd640a6ae5a94eea9467032fbee8a89b P a377ebe96e7c42f38ba604cbccdebfaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a377ebe96e7c42f38ba604cbccdebfaa, State: Initialized, Role: FOLLOWER
I20260929 16:26:00.419739 2017 consensus_queue.cc:260] T dd640a6ae5a94eea9467032fbee8a89b P a377ebe96e7c42f38ba604cbccdebfaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } }
I20260929 16:26:00.419989 2017 ts_tablet_manager.cc:1428] T dd640a6ae5a94eea9467032fbee8a89b P a377ebe96e7c42f38ba604cbccdebfaa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:26:00.420084 2035 tablet_replica.cc:439] T dd640a6ae5a94eea9467032fbee8a89b P a377ebe96e7c42f38ba604cbccdebfaa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } }
I20260929 16:26:00.420267 2035 tablet_replica.cc:442] T dd640a6ae5a94eea9467032fbee8a89b P a377ebe96e7c42f38ba604cbccdebfaa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:26:00.420617 2018 tablet_bootstrap.cc:492] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0: No bootstrap required, opened a new log
I20260929 16:26:00.420722 2018 ts_tablet_manager.cc:1397] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260929 16:26:00.421185 2018 raft_consensus.cc:348] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } }
I20260929 16:26:00.421345 2018 raft_consensus.cc:374] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:00.421379 2018 raft_consensus.cc:729] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5fa3284cff4be68ddf362d78991ce0, State: Initialized, Role: FOLLOWER
I20260929 16:26:00.421517 2018 consensus_queue.cc:260] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } }
I20260929 16:26:00.421815 2024 tablet_replica.cc:439] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } }
I20260929 16:26:00.422119 2024 tablet_replica.cc:442] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:26:00.423218 2018 ts_tablet_manager.cc:1428] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 16:26:00.640004 2024 raft_consensus.cc:482] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:26:00.640199 2024 raft_consensus.cc:504] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } }
I20260929 16:26:00.641518 2024 leader_election.cc:288] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a377ebe96e7c42f38ba604cbccdebfaa (127.0.98.2:38323), a6a6bd00bef7457983c31d7d81fe1855 (127.0.98.3:35015)
I20260929 16:26:00.645805 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.641931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51433 (local address 127.0.98.2:38323)
0929 16:26:00.642070 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:00.642076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:00.642335 (+ 259us) server_negotiation.cc:407] Connection header received
0929 16:26:00.642446 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:00.642452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:00.642515 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:00.642598 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:00.642957 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.643496 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.644692 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.645139 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.645289 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.645362 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:00.645555 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 16:26:00.645650 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:26:00.645869 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.641823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38323 (local address 127.0.98.1:51433)
0929 16:26:00.642229 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:00.642242 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:00.642350 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:00.642627 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:00.642630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:00.642648 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:00.642795 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:00.642803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.643612 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.643616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:00.644548 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:00.644557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.645287 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.645290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:00.645408 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.645432 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:26:00.645542 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 16:26:00.645641 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":1,"thread_start_us":134,"threads_started":1}
I20260929 16:26:00.645854 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.642034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38719 (local address 127.0.98.3:35015)
0929 16:26:00.642236 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:00.642240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:00.642532 (+ 292us) server_negotiation.cc:407] Connection header received
0929 16:26:00.642629 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:00.642631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:00.642675 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:00.642749 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:00.643079 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.643605 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.644805 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.645290 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.645398 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.645463 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:00.645668 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 16:26:00.645746 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 16:26:00.646263 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.641823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35015 (local address 127.0.98.1:38719)
0929 16:26:00.642349 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:00.642365 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:00.642517 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:00.642781 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:00.642783 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:00.642797 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:00.642928 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:00.642936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.643746 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.643749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:00.644674 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:00.644685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.645410 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.645414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:00.645534 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.645556 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:26:00.645634 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 16:26:00.645698 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":83,"thread_start_us":106,"threads_started":1}
I20260929 16:26:00.646327 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd640a6ae5a94eea9467032fbee8a89b" candidate_uuid: "4f5fa3284cff4be68ddf362d78991ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" is_pre_election: true
I20260929 16:26:00.646557 1807 raft_consensus.cc:2454] T dd640a6ae5a94eea9467032fbee8a89b P a377ebe96e7c42f38ba604cbccdebfaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5fa3284cff4be68ddf362d78991ce0 in term 0.
I20260929 16:26:00.646790 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd640a6ae5a94eea9467032fbee8a89b" candidate_uuid: "4f5fa3284cff4be68ddf362d78991ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a6bd00bef7457983c31d7d81fe1855" is_pre_election: true
I20260929 16:26:00.646934 1947 raft_consensus.cc:2454] T dd640a6ae5a94eea9467032fbee8a89b P a6a6bd00bef7457983c31d7d81fe1855 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5fa3284cff4be68ddf362d78991ce0 in term 0.
I20260929 16:26:00.646942 1601 leader_election.cc:302] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f5fa3284cff4be68ddf362d78991ce0, a377ebe96e7c42f38ba604cbccdebfaa; no voters:
I20260929 16:26:00.647255 2024 raft_consensus.cc:2781] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:26:00.647392 2024 raft_consensus.cc:482] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:26:00.647441 2024 raft_consensus.cc:3037] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:00.648514 2024 raft_consensus.cc:504] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } }
I20260929 16:26:00.648908 2024 leader_election.cc:288] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0 [CANDIDATE]: Term 1 election: Requested vote from peers a377ebe96e7c42f38ba604cbccdebfaa (127.0.98.2:38323), a6a6bd00bef7457983c31d7d81fe1855 (127.0.98.3:35015)
I20260929 16:26:00.649367 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd640a6ae5a94eea9467032fbee8a89b" candidate_uuid: "4f5fa3284cff4be68ddf362d78991ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a6bd00bef7457983c31d7d81fe1855"
I20260929 16:26:00.649372 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd640a6ae5a94eea9467032fbee8a89b" candidate_uuid: "4f5fa3284cff4be68ddf362d78991ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a377ebe96e7c42f38ba604cbccdebfaa"
I20260929 16:26:00.649526 1947 raft_consensus.cc:3037] T dd640a6ae5a94eea9467032fbee8a89b P a6a6bd00bef7457983c31d7d81fe1855 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:00.649549 1807 raft_consensus.cc:3037] T dd640a6ae5a94eea9467032fbee8a89b P a377ebe96e7c42f38ba604cbccdebfaa [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:00.650547 1947 raft_consensus.cc:2454] T dd640a6ae5a94eea9467032fbee8a89b P a6a6bd00bef7457983c31d7d81fe1855 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5fa3284cff4be68ddf362d78991ce0 in term 1.
I20260929 16:26:00.650549 1807 raft_consensus.cc:2454] T dd640a6ae5a94eea9467032fbee8a89b P a377ebe96e7c42f38ba604cbccdebfaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5fa3284cff4be68ddf362d78991ce0 in term 1.
I20260929 16:26:00.650952 1598 leader_election.cc:302] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f5fa3284cff4be68ddf362d78991ce0, a6a6bd00bef7457983c31d7d81fe1855; no voters:
I20260929 16:26:00.651191 2024 raft_consensus.cc:2781] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:00.651463 2024 raft_consensus.cc:686] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0 [term 1 LEADER]: Becoming Leader. State: Replica: 4f5fa3284cff4be68ddf362d78991ce0, State: Running, Role: LEADER
I20260929 16:26:00.651762 2024 consensus_queue.cc:237] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } }
I20260929 16:26:00.653374 2061 tablet_replica.cc:439] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0: TxnStatusTablet state changed. Reason: New leader 4f5fa3284cff4be68ddf362d78991ce0. Latest consensus state: current_term: 1 leader_uuid: "4f5fa3284cff4be68ddf362d78991ce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } }
I20260929 16:26:00.653522 2061 tablet_replica.cc:442] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:26:00.653903 2063 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 16:26:00.654078 2063 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 16:26:00.654520 1506 catalog_manager.cc:5495] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f5fa3284cff4be68ddf362d78991ce0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4f5fa3284cff4be68ddf362d78991ce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } health_report { overall_health: UNKNOWN } } }
I20260929 16:26:00.675715 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.671890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51366 (local address 127.0.98.1:40661)
0929 16:26:00.672026 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:00.672030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:00.672045 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:26:00.672224 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:00.672227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:00.672283 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:00.672397 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:00.672657 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.673190 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.674143 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.674377 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.674494 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.674815 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:00.675285 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:00.675421 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:00.675502 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 16:26:00.675563 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:26:00.675719 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.671769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40661 (local address 127.0.0.1:51366)
0929 16:26:00.671913 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:00.671929 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:00.672048 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:00.672414 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:00.672417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:00.672426 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:26:00.672545 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:00.672551 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.673374 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.673377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:00.674005 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:00.674015 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.674498 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.674501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:00.674605 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.674656 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:26:00.675399 (+ 743us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:26:00.675419 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 16:26:00.675493 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 16:26:00.675601 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 16:26:00.678969 1947 raft_consensus.cc:1261] T dd640a6ae5a94eea9467032fbee8a89b P a6a6bd00bef7457983c31d7d81fe1855 [term 1 FOLLOWER]: Refusing update from remote peer 4f5fa3284cff4be68ddf362d78991ce0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:00.679145 1807 raft_consensus.cc:1261] T dd640a6ae5a94eea9467032fbee8a89b P a377ebe96e7c42f38ba604cbccdebfaa [term 1 FOLLOWER]: Refusing update from remote peer 4f5fa3284cff4be68ddf362d78991ce0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:00.679562 2024 consensus_queue.cc:1035] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:00.679798 2061 consensus_queue.cc:1035] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:00.681805 2035 tablet_replica.cc:439] T dd640a6ae5a94eea9467032fbee8a89b P a377ebe96e7c42f38ba604cbccdebfaa: TxnStatusTablet state changed. Reason: New leader 4f5fa3284cff4be68ddf362d78991ce0. Latest consensus state: current_term: 1 leader_uuid: "4f5fa3284cff4be68ddf362d78991ce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } }
I20260929 16:26:00.681953 2035 tablet_replica.cc:442] T dd640a6ae5a94eea9467032fbee8a89b P a377ebe96e7c42f38ba604cbccdebfaa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:26:00.683751 2022 tablet_replica.cc:439] T dd640a6ae5a94eea9467032fbee8a89b P a6a6bd00bef7457983c31d7d81fe1855: TxnStatusTablet state changed. Reason: New leader 4f5fa3284cff4be68ddf362d78991ce0. Latest consensus state: current_term: 1 leader_uuid: "4f5fa3284cff4be68ddf362d78991ce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } }
I20260929 16:26:00.683809 2061 tablet_replica.cc:439] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f5fa3284cff4be68ddf362d78991ce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } }
I20260929 16:26:00.683948 2061 tablet_replica.cc:442] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:26:00.683951 2022 tablet_replica.cc:442] T dd640a6ae5a94eea9467032fbee8a89b P a6a6bd00bef7457983c31d7d81fe1855: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:26:00.684724 2022 tablet_replica.cc:439] T dd640a6ae5a94eea9467032fbee8a89b P a6a6bd00bef7457983c31d7d81fe1855: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f5fa3284cff4be68ddf362d78991ce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } }
I20260929 16:26:00.684834 2022 tablet_replica.cc:442] T dd640a6ae5a94eea9467032fbee8a89b P a6a6bd00bef7457983c31d7d81fe1855: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:26:00.687608 2024 tablet_replica.cc:439] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f5fa3284cff4be68ddf362d78991ce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } }
I20260929 16:26:00.687752 2024 tablet_replica.cc:442] T dd640a6ae5a94eea9467032fbee8a89b P 4f5fa3284cff4be68ddf362d78991ce0: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:26:00.689059 2035 tablet_replica.cc:439] T dd640a6ae5a94eea9467032fbee8a89b P a377ebe96e7c42f38ba604cbccdebfaa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f5fa3284cff4be68ddf362d78991ce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 } } peers { permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "a6a6bd00bef7457983c31d7d81fe1855" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35015 } } }
I20260929 16:26:00.689189 2035 tablet_replica.cc:442] T dd640a6ae5a94eea9467032fbee8a89b P a377ebe96e7c42f38ba604cbccdebfaa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:26:00.689651 2030 mvcc.cc:205] Tried to move back new op lower bound from 7334703762135224320 to 7334703762032189440. Current Snapshot: MvccSnapshot[applied={T|T < 7334703762135224320 or (T in {7334703762135224320})}]
I20260929 16:26:00.695729 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.691328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44936 (local address 127.0.98.3:35015)
0929 16:26:00.691453 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:00.691458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:00.691576 (+ 118us) server_negotiation.cc:407] Connection header received
0929 16:26:00.691781 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:00.691784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:00.691840 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:00.691937 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:00.692199 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.692853 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.694187 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.694412 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.694498 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.694774 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:00.695293 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:00.695386 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:00.695504 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 16:26:00.695557 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:26:00.708639 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.704259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34431 (local address 127.0.98.1:40661)
0929 16:26:00.704412 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:00.704417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:00.704653 (+ 236us) server_negotiation.cc:407] Connection header received
0929 16:26:00.704799 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:00.704802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:00.704850 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:00.704953 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:00.705286 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.705867 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.707187 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.707624 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.707731 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.708119 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:00.708207 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:00.708319 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:00.708474 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 16:26:00.708525 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:26:00.708698 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.704179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40661 (local address 127.0.98.3:34431)
0929 16:26:00.704551 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:00.704568 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:00.704698 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:00.704990 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:00.704993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:00.705001 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:26:00.705149 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:00.705156 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.706003 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.706007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:00.707056 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:00.707068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.707755 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.707760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:00.707871 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.707943 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:26:00.708326 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:26:00.708333 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 16:26:00.708415 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 16:26:00.708511 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":145,"threads_started":1}
I20260929 16:26:00.716775 1807 raft_consensus.cc:1261] T 8f3d771af60f4e6792e269ea945d2f1a P a377ebe96e7c42f38ba604cbccdebfaa [term 1 FOLLOWER]: Refusing update from remote peer a6a6bd00bef7457983c31d7d81fe1855: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:00.717116 1667 raft_consensus.cc:1261] T 8f3d771af60f4e6792e269ea945d2f1a P 4f5fa3284cff4be68ddf362d78991ce0 [term 1 FOLLOWER]: Refusing update from remote peer a6a6bd00bef7457983c31d7d81fe1855: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:00.717367 2022 consensus_queue.cc:1035] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a377ebe96e7c42f38ba604cbccdebfaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:00.717661 2047 consensus_queue.cc:1035] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5fa3284cff4be68ddf362d78991ce0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:00.745381 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.741009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42853 (local address 127.0.98.3:35015)
0929 16:26:00.741149 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:00.741154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:00.741436 (+ 282us) server_negotiation.cc:407] Connection header received
0929 16:26:00.741597 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:00.741600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:00.741658 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:00.741739 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:00.742107 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.742971 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.744015 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.744286 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.744422 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.744761 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:00.744888 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:00.744970 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:00.745134 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 16:26:00.745235 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:26:00.745409 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.740819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35015 (local address 127.0.98.1:42853)
0929 16:26:00.741317 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:00.741335 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:00.741483 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:00.741780 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:00.741786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:00.741798 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:26:00.741964 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:00.741971 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.743093 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.743101 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:00.743861 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:00.743871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.744426 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.744431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:00.744545 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.744600 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:26:00.744996 (+ 396us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:26:00.745004 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 16:26:00.745107 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 16:26:00.745192 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":127,"threads_started":1}
I20260929 16:26:00.780205 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:00.780397 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:00.783068 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87d46e4029ef47d2b0890bb954a13ad7: Sending DeleteTablet for 3 replicas of tablet 8f3d771af60f4e6792e269ea945d2f1a
I20260929 16:26:00.783926 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f3d771af60f4e6792e269ea945d2f1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51362
I20260929 16:26:00.784166 2041 tablet_replica.cc:330] T 8f3d771af60f4e6792e269ea945d2f1a P 4f5fa3284cff4be68ddf362d78991ce0: stopping tablet replica
I20260929 16:26:00.784180 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f3d771af60f4e6792e269ea945d2f1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51644
I20260929 16:26:00.784188 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f3d771af60f4e6792e269ea945d2f1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44932
I20260929 16:26:00.784301 2041 raft_consensus.cc:2229] T 8f3d771af60f4e6792e269ea945d2f1a P 4f5fa3284cff4be68ddf362d78991ce0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:00.784443 2041 raft_consensus.cc:2258] T 8f3d771af60f4e6792e269ea945d2f1a P 4f5fa3284cff4be68ddf362d78991ce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:00.784621 2042 tablet_replica.cc:330] T 8f3d771af60f4e6792e269ea945d2f1a P a377ebe96e7c42f38ba604cbccdebfaa: stopping tablet replica
I20260929 16:26:00.784749 2042 raft_consensus.cc:2229] T 8f3d771af60f4e6792e269ea945d2f1a P a377ebe96e7c42f38ba604cbccdebfaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:00.784871 2042 raft_consensus.cc:2258] T 8f3d771af60f4e6792e269ea945d2f1a P a377ebe96e7c42f38ba604cbccdebfaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:00.785048 2040 tablet_replica.cc:330] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855: stopping tablet replica
I20260929 16:26:00.785171 2040 raft_consensus.cc:2229] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:26:00.785534 2040 raft_consensus.cc:2258] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:00.786347 2040 ts_tablet_manager.cc:1905] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:00.786347 2042 ts_tablet_manager.cc:1905] T 8f3d771af60f4e6792e269ea945d2f1a P a377ebe96e7c42f38ba604cbccdebfaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:00.789403 2040 ts_tablet_manager.cc:1918] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:26:00.789409 2042 ts_tablet_manager.cc:1918] T 8f3d771af60f4e6792e269ea945d2f1a P a377ebe96e7c42f38ba604cbccdebfaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:26:00.789510 2040 log.cc:1196] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/8f3d771af60f4e6792e269ea945d2f1a
I20260929 16:26:00.789510 2042 log.cc:1196] T 8f3d771af60f4e6792e269ea945d2f1a P a377ebe96e7c42f38ba604cbccdebfaa: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/8f3d771af60f4e6792e269ea945d2f1a
I20260929 16:26:00.789784 2042 ts_tablet_manager.cc:1939] T 8f3d771af60f4e6792e269ea945d2f1a P a377ebe96e7c42f38ba604cbccdebfaa: Deleting consensus metadata
I20260929 16:26:00.789784 2040 ts_tablet_manager.cc:1939] T 8f3d771af60f4e6792e269ea945d2f1a P a6a6bd00bef7457983c31d7d81fe1855: Deleting consensus metadata
I20260929 16:26:00.790490 2041 ts_tablet_manager.cc:1905] T 8f3d771af60f4e6792e269ea945d2f1a P 4f5fa3284cff4be68ddf362d78991ce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:00.790870 1494 catalog_manager.cc:4841] TS a6a6bd00bef7457983c31d7d81fe1855 (127.0.98.3:35015): tablet 8f3d771af60f4e6792e269ea945d2f1a (table test-table [id=d214e13ac6b740faa49f8d7f9d0f2c17]) successfully deleted
I20260929 16:26:00.793678 2041 ts_tablet_manager.cc:1918] T 8f3d771af60f4e6792e269ea945d2f1a P 4f5fa3284cff4be68ddf362d78991ce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:26:00.793793 2041 log.cc:1196] T 8f3d771af60f4e6792e269ea945d2f1a P 4f5fa3284cff4be68ddf362d78991ce0: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790699152396020-392-0/minicluster-data/ts-0/wal/wals/8f3d771af60f4e6792e269ea945d2f1a
I20260929 16:26:00.794044 2041 ts_tablet_manager.cc:1939] T 8f3d771af60f4e6792e269ea945d2f1a P 4f5fa3284cff4be68ddf362d78991ce0: Deleting consensus metadata
I20260929 16:26:00.794888 1495 catalog_manager.cc:4841] TS 4f5fa3284cff4be68ddf362d78991ce0 (127.0.98.1:40661): tablet 8f3d771af60f4e6792e269ea945d2f1a (table test-table [id=d214e13ac6b740faa49f8d7f9d0f2c17]) successfully deleted
I20260929 16:26:00.795351 1497 catalog_manager.cc:4841] TS a377ebe96e7c42f38ba604cbccdebfaa (127.0.98.2:38323): tablet 8f3d771af60f4e6792e269ea945d2f1a (table test-table [id=d214e13ac6b740faa49f8d7f9d0f2c17]) successfully deleted
Sep 29 16:26:00 dist-test-slave-vq9v krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699159, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 16:26:00.806804 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.795618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51368 (local address 127.0.98.1:40661)
0929 16:26:00.795751 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:00.795756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:00.796225 (+ 469us) server_negotiation.cc:407] Connection header received
0929 16:26:00.796388 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:00.796391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:00.796446 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:00.796539 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:00.797288 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.797796 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.798669 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.798955 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.799073 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.802534 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:00.802546 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:00.802561 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:00.802594 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:00.804491 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:00.805048 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:00.805052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:00.805054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:00.805149 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:00.805684 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:00.805687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:00.805689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:00.806032 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:00.806275 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:00.806484 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 16:26:00.806620 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 16:26:00.807508 1707 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51368
I20260929 16:26:00.818730 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:00.809804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36196 (local address 127.0.98.62:34879)
0929 16:26:00.809953 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:00.809958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:00.810213 (+ 255us) server_negotiation.cc:407] Connection header received
0929 16:26:00.810389 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:00.810392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:00.810442 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:00.810533 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:00.811240 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.811798 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.812636 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:00.812876 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:00.812994 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:00.814626 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:00.814630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:00.814633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:00.814784 (+ 151us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:00.816820 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:00.817371 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:00.817375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:00.817376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:00.817465 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:00.817770 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:00.817773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:00.817775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:00.817982 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:00.818136 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:00.818399 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 16:26:00.818539 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 16:26:00.819160 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36196
I20260929 16:26:00.820847 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 1582
I20260929 16:26:00.831564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 1722
I20260929 16:26:00.843412 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 1862
I20260929 16:26:00.849829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 1478
2026-09-29T16:26:00Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3196 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:26:00 dist-test-slave-vq9v krb5kdc[2086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:26:00 dist-test-slave-vq9v krb5kdc[2086](info): set up 2 sockets
Sep 29 16:26:00 dist-test-slave-vq9v krb5kdc[2086](info): commencing operation
krb5kdc: starting...
W20260929 16:26:01.928269 392 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260929 16:26:02.061508 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.133s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260929 16:26:02.280867 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.219s user 0.002s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:26:02 dist-test-slave-vq9v krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:26:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:26:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:26:02.396849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34265
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:02.514539 2099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:02.514825 2099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:02.514868 2099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:02.518965 2099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:26:02.519030 2099 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 16:26:02.519047 2099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:02.519066 2099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:02.519083 2099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:26:02.519098 2099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:26:02.524554 2099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34265
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:02.525689 2099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:02.526671 2099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:02.535663 2107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:02.535666 2104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:02.535722 2099 server_base.cc:1034] running on GCE node
W20260929 16:26:02.535724 2105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:02.536510 2099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:02.537887 2099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:02.539126 2099 hybrid_clock.cc:648] HybridClock initialized: now 1790699162539091 us; error 50 us; skew 500 ppm
Sep 29 16:26:02 dist-test-slave-vq9v krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:02.542126 2099 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:26:02.543135 2099 webserver.cc:460] Webserver started at http://127.0.98.62:45687/ using document root <none> and password file <none>
I20260929 16:26:02.543715 2099 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:02.543807 2099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:02.544100 2099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:02.546425 2099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "9593a903c18044deb7e38a0c478e601a"
format_stamp: "Formatted at 2026-09-29 16:26:02 on dist-test-slave-vq9v"
server_key: "04bad0ba28eab7e7923dc9e171ee0f9c"
server_key_iv: "1be453bd38d51739f955c63b00a67940"
server_key_version: "encryptionkey@0"
I20260929 16:26:02.547016 2099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "9593a903c18044deb7e38a0c478e601a"
format_stamp: "Formatted at 2026-09-29 16:26:02 on dist-test-slave-vq9v"
server_key: "04bad0ba28eab7e7923dc9e171ee0f9c"
server_key_iv: "1be453bd38d51739f955c63b00a67940"
server_key_version: "encryptionkey@0"
I20260929 16:26:02.551100 2099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:26:02.554044 2114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:02.555500 2099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:02.555688 2099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "9593a903c18044deb7e38a0c478e601a"
format_stamp: "Formatted at 2026-09-29 16:26:02 on dist-test-slave-vq9v"
server_key: "04bad0ba28eab7e7923dc9e171ee0f9c"
server_key_iv: "1be453bd38d51739f955c63b00a67940"
server_key_version: "encryptionkey@0"
I20260929 16:26:02.555831 2099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:02.587952 2099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:02.591782 2099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:02.592109 2099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:02.601078 2099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34265
I20260929 16:26:02.601110 2176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34265 every 8 connection(s)
I20260929 16:26:02.602141 2099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:26:02.604055 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 2099
I20260929 16:26:02.604168 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:26:02.604455 392 external_mini_cluster.cc:1428] Setting key 2e90fa9002c09dcdb817e3cb5bc425b6
I20260929 16:26:02.605468 2177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:02.611891 2177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a: Bootstrap starting.
Sep 29 16:26:02 dist-test-slave-vq9v krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:02.614822 2177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:02.616221 2177 log.cc:824] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:02.618891 2177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a: No bootstrap required, opened a new log
I20260929 16:26:02.620187 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:02.605924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56728 (local address 127.0.98.62:34265)
0929 16:26:02.606341 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:02.606356 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:02.606391 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:26:02.607024 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:02.607039 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:02.607383 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:02.607655 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:02.608585 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:02.609722 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:02.610970 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:02.611553 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:02.611696 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:02.614601 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:02.614607 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:02.614625 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:02.614661 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:02.616738 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:02.617382 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:02.617388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:02.617395 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:02.617532 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:02.617835 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:02.617839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:02.617840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:02.618347 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:02.618538 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:02.618871 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 16:26:02.619236 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":140,"threads_started":1}
I20260929 16:26:02.622598 2177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9593a903c18044deb7e38a0c478e601a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34265 } }
I20260929 16:26:02.622859 2177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:02.622898 2177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9593a903c18044deb7e38a0c478e601a, State: Initialized, Role: FOLLOWER
I20260929 16:26:02.623320 2177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9593a903c18044deb7e38a0c478e601a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34265 } }
I20260929 16:26:02.623452 2177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:26:02.623493 2177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:26:02.623575 2177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:02.624442 2177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9593a903c18044deb7e38a0c478e601a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34265 } }
I20260929 16:26:02.624742 2177 leader_election.cc:302] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9593a903c18044deb7e38a0c478e601a; no voters:
I20260929 16:26:02.625000 2177 leader_election.cc:288] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:26:02.625247 2182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:02.625502 2182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a [term 1 LEADER]: Becoming Leader. State: Replica: 9593a903c18044deb7e38a0c478e601a, State: Running, Role: LEADER
I20260929 16:26:02.625845 2182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9593a903c18044deb7e38a0c478e601a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34265 } }
I20260929 16:26:02.626317 2177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:26:02.627934 2183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9593a903c18044deb7e38a0c478e601a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9593a903c18044deb7e38a0c478e601a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34265 } } }
I20260929 16:26:02.628078 2183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:02.628903 2184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9593a903c18044deb7e38a0c478e601a. Latest consensus state: current_term: 1 leader_uuid: "9593a903c18044deb7e38a0c478e601a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9593a903c18044deb7e38a0c478e601a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34265 } } }
I20260929 16:26:02.629019 2184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a [sys.catalog]: This master's current role is: LEADER
W20260929 16:26:02.631584 2198 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:26:02.631657 2198 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:26:02.631758 2191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:26:02.632625 2191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:26:02.639001 2191 catalog_manager.cc:1347] Generated new cluster ID: 4e6b9f6fd27a45ed916dfcc11e30a0d7
I20260929 16:26:02.639096 2191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:26:02.663450 2191 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:26:02.664754 2191 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:26:02.671008 2191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a: Generated new TSK 0
I20260929 16:26:02.671831 2191 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 16:26:02.717264 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34265
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:02.835269 2203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:02.835585 2203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:02.835638 2203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:02.839905 2203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:02.839998 2203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:02.840044 2203 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 16:26:02.840061 2203 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:26:02.840157 2203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:26:02.845916 2203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:02.847112 2203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:02.848233 2203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:02.858711 2208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:02.858711 2211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:02.858757 2209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:02.859028 2203 server_base.cc:1034] running on GCE node
I20260929 16:26:02.859550 2203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:02.860400 2203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:02.861584 2203 hybrid_clock.cc:648] HybridClock initialized: now 1790699162861569 us; error 32 us; skew 500 ppm
Sep 29 16:26:02 dist-test-slave-vq9v krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:02.864542 2203 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:26:02.865612 2203 webserver.cc:460] Webserver started at http://127.0.98.1:36853/ using document root <none> and password file <none>
I20260929 16:26:02.866145 2203 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:02.866204 2203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:02.866365 2203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:02.868403 2203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "2a7e4632bf4d43df9a0d3f0011c522ac"
format_stamp: "Formatted at 2026-09-29 16:26:02 on dist-test-slave-vq9v"
server_key: "8c278346e526e2f7c130e225b06fc78a"
server_key_iv: "a8a92413ee1a08f368013fa5bfda3540"
server_key_version: "encryptionkey@0"
I20260929 16:26:02.868953 2203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2a7e4632bf4d43df9a0d3f0011c522ac"
format_stamp: "Formatted at 2026-09-29 16:26:02 on dist-test-slave-vq9v"
server_key: "8c278346e526e2f7c130e225b06fc78a"
server_key_iv: "a8a92413ee1a08f368013fa5bfda3540"
server_key_version: "encryptionkey@0"
I20260929 16:26:02.872999 2203 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:26:02.875651 2218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:02.876957 2203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:26:02.877056 2203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "2a7e4632bf4d43df9a0d3f0011c522ac"
format_stamp: "Formatted at 2026-09-29 16:26:02 on dist-test-slave-vq9v"
server_key: "8c278346e526e2f7c130e225b06fc78a"
server_key_iv: "a8a92413ee1a08f368013fa5bfda3540"
server_key_version: "encryptionkey@0"
I20260929 16:26:02.877251 2203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:02.899286 2203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:02.902738 2203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:02.902953 2203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:02.904459 2203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:02.904518 2203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:02.904559 2203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:02.904614 2203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:26:02 dist-test-slave-vq9v krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:02.916342 2203 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44297
I20260929 16:26:02.916410 2333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44297 every 8 connection(s)
I20260929 16:26:02.917474 2203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:26:02.919294 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:02.905454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60775 (local address 127.0.98.62:34265)
0929 16:26:02.905609 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:02.905614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:02.906726 (+ 1112us) server_negotiation.cc:407] Connection header received
0929 16:26:02.907859 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:02.907864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:02.907931 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:02.908040 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:02.909535 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:02.910125 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:02.911328 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:02.911644 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:02.911765 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:02.914925 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:02.914929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:02.914932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:02.914961 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:02.916732 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:02.917453 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:02.917480 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:02.917481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:02.917635 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:02.918090 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:02.918096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:02.918100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:02.918361 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:02.918486 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:02.918985 (+ 499us) server_negotiation.cc:299] Negotiation successful
0929 16:26:02.919124 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:26:02.920046 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:02.906021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34265 (local address 127.0.98.1:60775)
0929 16:26:02.906554 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:02.906606 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:02.907599 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:02.908254 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:02.908261 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:02.908732 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:02.909375 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:02.909412 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:02.910248 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:02.910252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:02.911197 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:02.911206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:02.911812 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:02.911817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:02.912006 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:02.912719 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:02.912738 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:02.914720 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:02.916840 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:02.916847 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:02.916861 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:02.917306 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:02.917758 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:02.917764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:02.917768 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:02.917971 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:02.918482 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:02.918488 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:02.918735 (+ 247us) client_negotiation.cc:769] Sending connection context
0929 16:26:02.918921 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 16:26:02.919238 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":141,"threads_started":1}
I20260929 16:26:02.921612 2334 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34265
I20260929 16:26:02.921770 2334 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:02.922276 2334 heartbeater.cc:511] Master 127.0.98.62:34265 requested a full tablet report, sending...
I20260929 16:26:02.923983 2130 ts_manager.cc:194] Registered new tserver with Master: 2a7e4632bf4d43df9a0d3f0011c522ac (127.0.98.1:44297)
I20260929 16:26:02.924688 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 2203
I20260929 16:26:02.924815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:26:02.925081 392 external_mini_cluster.cc:1428] Setting key a60da96ccf0cc8ddeb1ac80f9a45eda0
I20260929 16:26:02.925616 2130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60775
I20260929 16:26:02.932128 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:02.923861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56533 (local address 127.0.98.62:34265)
0929 16:26:02.923992 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:02.924002 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:02.924219 (+ 217us) server_negotiation.cc:407] Connection header received
0929 16:26:02.924259 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:02.924261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:02.924301 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:02.924368 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:02.924905 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:02.925727 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:02.926952 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:02.927180 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:02.927294 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:02.928659 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:02.928662 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:02.928664 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:02.928686 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:02.930264 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:02.930811 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:02.930813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:02.930814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:02.930888 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:02.931190 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:02.931192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:02.931193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:02.931366 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:02.931442 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:02.931805 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 16:26:02.931953 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":52}
I20260929 16:26:02.932271 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:02.923726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34265 (local address 127.0.98.1:56533)
0929 16:26:02.924054 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:02.924071 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:02.924182 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:02.924410 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:02.924412 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:02.924643 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:02.924769 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:02.924777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:02.925894 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:02.925897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:02.926835 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:02.926844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:02.927296 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:02.927300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:02.927421 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:02.927987 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:02.927999 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:02.928516 (+ 517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:02.930388 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:02.930395 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:02.930400 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:02.930705 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:02.930986 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:02.930990 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:02.930992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:02.931096 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:02.931473 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:02.931479 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:02.931619 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 16:26:02.931747 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 16:26:02.931956 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":3,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 16:26:02.957823 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34265
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:03.085161 2343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:03.085578 2343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:03.085632 2343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:03.090158 2343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:03.090247 2343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:03.090291 2343 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 16:26:03.090306 2343 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:26:03.090376 2343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:26:03.096117 2343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:03.097397 2343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:03.098466 2343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:03.109377 2348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:03.109377 2351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:03.109400 2349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:03.109792 2343 server_base.cc:1034] running on GCE node
I20260929 16:26:03.110396 2343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:03.111081 2343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:03.112272 2343 hybrid_clock.cc:648] HybridClock initialized: now 1790699163112254 us; error 34 us; skew 500 ppm
Sep 29 16:26:03 dist-test-slave-vq9v krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:03.115444 2343 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:26:03.116446 2343 webserver.cc:460] Webserver started at http://127.0.98.2:46153/ using document root <none> and password file <none>
I20260929 16:26:03.117014 2343 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:03.117079 2343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:03.117358 2343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:03.119656 2343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "469c940f7a7d40328acb8639ea30596d"
format_stamp: "Formatted at 2026-09-29 16:26:03 on dist-test-slave-vq9v"
server_key: "ba838d3493775cad8eacd936feaf69bd"
server_key_iv: "e735ae66b27eb5463026b99b2a623535"
server_key_version: "encryptionkey@0"
I20260929 16:26:03.120252 2343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "469c940f7a7d40328acb8639ea30596d"
format_stamp: "Formatted at 2026-09-29 16:26:03 on dist-test-slave-vq9v"
server_key: "ba838d3493775cad8eacd936feaf69bd"
server_key_iv: "e735ae66b27eb5463026b99b2a623535"
server_key_version: "encryptionkey@0"
I20260929 16:26:03.124779 2343 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260929 16:26:03.127846 2358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:03.129532 2343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:03.129657 2343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "469c940f7a7d40328acb8639ea30596d"
format_stamp: "Formatted at 2026-09-29 16:26:03 on dist-test-slave-vq9v"
server_key: "ba838d3493775cad8eacd936feaf69bd"
server_key_iv: "e735ae66b27eb5463026b99b2a623535"
server_key_version: "encryptionkey@0"
I20260929 16:26:03.129818 2343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:03.159334 2343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:03.162892 2343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:03.163154 2343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:03.164665 2343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:03.164750 2343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:03.164812 2343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:03.164863 2343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:03.176074 2343 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38023
I20260929 16:26:03.176147 2473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38023 every 8 connection(s)
Sep 29 16:26:03 dist-test-slave-vq9v krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:03.177289 2343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:26:03.181702 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.165835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54019 (local address 127.0.98.62:34265)
0929 16:26:03.166008 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:03.166013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:03.167221 (+ 1208us) server_negotiation.cc:407] Connection header received
0929 16:26:03.168694 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:03.168697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:03.168752 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:03.168841 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:03.170871 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.171725 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.173691 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.174039 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.174182 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.177435 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:03.177439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:03.177442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:03.177472 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:03.179322 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:03.179975 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:03.179978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:03.179980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:03.180081 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:03.180471 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:03.180474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:03.180476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:03.180678 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:03.180787 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:03.181438 (+ 651us) server_negotiation.cc:299] Negotiation successful
0929 16:26:03.181570 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:26:03.182662 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.166489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34265 (local address 127.0.98.2:54019)
0929 16:26:03.167011 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:03.167080 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:03.168401 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:03.169091 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:03.169104 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:03.169744 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:03.170615 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.170642 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.171878 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.171886 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.173529 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.173542 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.174237 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.174241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.174512 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.175196 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:03.175216 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:03.177164 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:03.179530 (+ 2366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:03.179537 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:03.179554 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:03.179839 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:03.180181 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:03.180185 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:03.180187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:03.180342 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:03.180798 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:03.180806 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:03.181075 (+ 269us) client_negotiation.cc:769] Sending connection context
0929 16:26:03.181309 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 16:26:03.181653 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":119,"threads_started":1}
I20260929 16:26:03.184150 2474 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34265
I20260929 16:26:03.184319 2474 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:03.184828 2474 heartbeater.cc:511] Master 127.0.98.62:34265 requested a full tablet report, sending...
I20260929 16:26:03.185813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 2343
I20260929 16:26:03.185942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:26:03.185977 2131 ts_manager.cc:194] Registered new tserver with Master: 469c940f7a7d40328acb8639ea30596d (127.0.98.2:38023)
I20260929 16:26:03.186239 392 external_mini_cluster.cc:1428] Setting key 90a9a71eb95d7687a486f31cd4854397
I20260929 16:26:03.186736 2131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54019
I20260929 16:26:03.194743 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.187130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60839 (local address 127.0.98.62:34265)
0929 16:26:03.187236 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:03.187241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:03.187256 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:26:03.187296 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:03.187299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:03.187344 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:03.187427 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:03.187951 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.188750 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.189806 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.190042 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.190155 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.191319 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:03.191323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:03.191325 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:03.191349 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:03.192881 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:03.193466 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:03.193471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:03.193473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:03.193553 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:03.193843 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:03.193846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:03.193848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:03.194073 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:03.194187 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:03.194465 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 16:26:03.194575 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 16:26:03.194794 2480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.186784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34265 (local address 127.0.98.2:60839)
0929 16:26:03.187116 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:03.187134 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:03.187227 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:03.187447 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:03.187450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:03.187683 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:03.187808 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.187816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.188909 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.188913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.189668 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.189677 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.190159 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.190162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.190258 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.190823 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:03.190834 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:03.191178 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:03.193059 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:03.193064 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:03.193067 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:03.193359 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:03.193654 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:03.193658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:03.193660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:03.193745 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:03.194176 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:03.194180 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:03.194282 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 16:26:03.194391 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 16:26:03.194563 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":130,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 16:26:03.217368 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34265
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:03.335186 2483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:03.335460 2483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:03.335510 2483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:03.339684 2483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:03.339761 2483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:03.339803 2483 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 16:26:03.339818 2483 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:26:03.339852 2483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:26:03.345490 2483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:03.346518 2483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:03.347488 2483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:26:03.357740 2483 server_base.cc:1034] running on GCE node
W20260929 16:26:03.357689 2489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:03.357722 2491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:03.357911 2488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:03.358381 2483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:03.359174 2483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:03.360412 2483 hybrid_clock.cc:648] HybridClock initialized: now 1790699163360378 us; error 54 us; skew 500 ppm
Sep 29 16:26:03 dist-test-slave-vq9v krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:03.363397 2483 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:26:03.364289 2483 webserver.cc:460] Webserver started at http://127.0.98.3:43753/ using document root <none> and password file <none>
I20260929 16:26:03.364876 2483 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:03.364969 2483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:03.365177 2483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:03.367337 2483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "4f645d147fa44b579755a51b53ff0664"
format_stamp: "Formatted at 2026-09-29 16:26:03 on dist-test-slave-vq9v"
server_key: "ad7c4a30acff606c87904ddc478bc7a9"
server_key_iv: "6703b0aa2f0895849988ca1f8d4e8fc0"
server_key_version: "encryptionkey@0"
I20260929 16:26:03.367913 2483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4f645d147fa44b579755a51b53ff0664"
format_stamp: "Formatted at 2026-09-29 16:26:03 on dist-test-slave-vq9v"
server_key: "ad7c4a30acff606c87904ddc478bc7a9"
server_key_iv: "6703b0aa2f0895849988ca1f8d4e8fc0"
server_key_version: "encryptionkey@0"
I20260929 16:26:03.371838 2483 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:26:03.374570 2498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:03.375952 2483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:26:03.376099 2483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "4f645d147fa44b579755a51b53ff0664"
format_stamp: "Formatted at 2026-09-29 16:26:03 on dist-test-slave-vq9v"
server_key: "ad7c4a30acff606c87904ddc478bc7a9"
server_key_iv: "6703b0aa2f0895849988ca1f8d4e8fc0"
server_key_version: "encryptionkey@0"
I20260929 16:26:03.376222 2483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:03.396983 2483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:03.400488 2483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:03.400727 2483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:03.402339 2483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:03.402436 2483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:03.402505 2483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:03.402552 2483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:26:03 dist-test-slave-vq9v krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:03.414024 2483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38465
I20260929 16:26:03.414131 2613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38465 every 8 connection(s)
I20260929 16:26:03.415138 2483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:26:03.418795 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.403583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47737 (local address 127.0.98.62:34265)
0929 16:26:03.403717 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:03.403721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:03.404980 (+ 1259us) server_negotiation.cc:407] Connection header received
0929 16:26:03.406413 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:03.406416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:03.406498 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:03.406624 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:03.408414 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.409432 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.411021 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.411279 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.411413 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.414812 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:03.414815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:03.414817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:03.414846 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:03.416545 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:03.417104 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:03.417108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:03.417109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:03.417285 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:03.417658 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:03.417660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:03.417661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:03.417857 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:03.417938 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:03.418474 (+ 536us) server_negotiation.cc:299] Negotiation successful
0929 16:26:03.418626 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:26:03.419620 2507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.404169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34265 (local address 127.0.98.3:47737)
0929 16:26:03.404726 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:03.404835 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:03.406146 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:03.406836 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:03.406844 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:03.407277 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:03.408154 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.408182 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.409588 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.409593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.410862 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.410874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.411472 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.411477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.411736 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.412551 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:03.412575 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:03.414574 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:03.416666 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:03.416671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:03.416683 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:03.416959 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:03.417401 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:03.417405 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:03.417407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:03.417560 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:03.417967 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:03.417972 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:03.418234 (+ 262us) client_negotiation.cc:769] Sending connection context
0929 16:26:03.418402 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 16:26:03.418685 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":144,"threads_started":1}
I20260929 16:26:03.421056 2614 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34265
I20260929 16:26:03.421245 2614 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:03.421784 2614 heartbeater.cc:511] Master 127.0.98.62:34265 requested a full tablet report, sending...
I20260929 16:26:03.423010 2130 ts_manager.cc:194] Registered new tserver with Master: 4f645d147fa44b579755a51b53ff0664 (127.0.98.3:38465)
I20260929 16:26:03.423758 2130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47737
I20260929 16:26:03.424373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 2483
I20260929 16:26:03.424485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:26:03.424796 392 external_mini_cluster.cc:1428] Setting key 8756601a86d54a46adba67f66da1ed83
I20260929 16:26:03.426883 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:26:03.431778 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.423500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48489 (local address 127.0.98.62:34265)
0929 16:26:03.423634 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:03.423639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:03.423862 (+ 223us) server_negotiation.cc:407] Connection header received
0929 16:26:03.423969 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:03.423971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:03.424035 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:03.424124 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:03.424760 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.425468 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.426576 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.426894 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.427055 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.428429 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:03.428432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:03.428435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:03.428461 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:03.430054 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:03.430517 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:03.430520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:03.430522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:03.430608 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:03.430896 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:03.430899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:03.430901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:03.431090 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:03.431200 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:03.431470 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 16:26:03.431614 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:26:03.431787 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.423419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34265 (local address 127.0.98.3:48489)
0929 16:26:03.423747 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:03.423763 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:03.423871 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:03.424242 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:03.424244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:03.424505 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:03.424626 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.424634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.425699 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.425701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.426408 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.426417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.427206 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.427208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.427280 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.427885 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:03.427897 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:03.428271 (+ 374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:03.430195 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:03.430199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:03.430201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:03.430408 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:03.430704 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:03.430708 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:03.430710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:03.430797 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:03.431189 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:03.431192 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:03.431295 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 16:26:03.431403 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 16:26:03.431588 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":121,"threads_started":1}
Sep 29 16:26:03 dist-test-slave-vq9v krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:26:03 dist-test-slave-vq9v krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:03.453527 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.445187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.0.98.62:34265)
0929 16:26:03.445386 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:03.445389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:03.445403 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:26:03.445446 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:03.445449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:03.445492 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:03.445565 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:03.446290 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.446810 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.447658 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.447867 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.447988 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.450481 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:03.450485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:03.450487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:03.450509 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:03.451834 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:03.452280 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:03.452284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:03.452286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:03.452364 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:03.452644 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:03.452649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:03.452650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:03.452833 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:03.452983 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:03.453243 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 16:26:03.453364 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260929 16:26:03.456708 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:26:03.459844 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:26:03.474902 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.469280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35410 (local address 127.0.98.3:38465)
0929 16:26:03.469770 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:03.469783 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:03.469807 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:26:03.469889 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:03.469894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:03.470136 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:03.470638 (+ 502us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:03.470802 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.471938 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.473537 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.474131 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.474273 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.474468 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:03.474626 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 16:26:03.474728 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":109,"threads_started":1}
I20260929 16:26:03.474949 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.469110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38023 (local address 127.0.0.1:58670)
0929 16:26:03.469908 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:03.469921 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:03.470004 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:03.470465 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:03.470468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:03.470485 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:03.470592 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.470602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.472104 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.472107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.473283 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.473293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.474552 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.474558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.474667 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.474694 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:26:03.474765 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 16:26:03.474825 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":1,"thread_start_us":132,"threads_started":1}
I20260929 16:26:03.475113 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.468472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44297 (local address 127.0.0.1:39060)
0929 16:26:03.468961 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:03.469009 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:03.469139 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:03.470306 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:03.470310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:03.470340 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:03.470514 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.470522 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.471725 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.471732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.473360 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.473372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.474382 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.474385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.474511 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.474551 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 16:26:03.474606 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 16:26:03.474729 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":3,"thread_start_us":90,"threads_started":1}
I20260929 16:26:03.474949 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.468670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38465 (local address 127.0.0.1:35410)
0929 16:26:03.469402 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:03.469418 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:03.469501 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:03.470302 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:03.470308 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:03.470331 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:03.470493 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.470500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.472071 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.472074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.473401 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.473412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.474268 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.474272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.474382 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.474467 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 16:26:03.474591 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 16:26:03.474760 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":83,"thread_start_us":132,"threads_started":1}
I20260929 16:26:03.475630 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.469332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58670 (local address 127.0.98.2:38023)
0929 16:26:03.469770 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:03.469783 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:03.470090 (+ 307us) server_negotiation.cc:407] Connection header received
0929 16:26:03.470164 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:03.470168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:03.470331 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:03.470479 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:03.470732 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.471938 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.473405 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.474239 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.474359 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.474520 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:03.475306 (+ 786us) server_negotiation.cc:299] Negotiation successful
0929 16:26:03.475446 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":128,"threads_started":1}
I20260929 16:26:03.475692 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.468668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39060 (local address 127.0.98.1:44297)
0929 16:26:03.469811 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:03.469818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:03.469842 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:26:03.469925 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:03.469930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:03.470146 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:03.470686 (+ 540us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:03.470822 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.471598 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.473497 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.474239 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.474848 (+ 609us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.475368 (+ 520us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:03.475414 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 16:26:03.475559 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1028,"thread_start_us":105,"threads_started":1}
I20260929 16:26:03.478314 2268 tablet_service.cc:1467] Processing CreateTablet for tablet 7b08e24ea0874d6f89b62f0104bda8f0 (DEFAULT_TABLE table=test-table [id=89b0af9609bf4f19a9cd46c60e5f07c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:03.478428 2541 tablet_service.cc:1467] Processing CreateTablet for tablet 7b08e24ea0874d6f89b62f0104bda8f0 (DEFAULT_TABLE table=test-table [id=89b0af9609bf4f19a9cd46c60e5f07c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:03.478428 2408 tablet_service.cc:1467] Processing CreateTablet for tablet 7b08e24ea0874d6f89b62f0104bda8f0 (DEFAULT_TABLE table=test-table [id=89b0af9609bf4f19a9cd46c60e5f07c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:03.479473 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b08e24ea0874d6f89b62f0104bda8f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:03.479516 2408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b08e24ea0874d6f89b62f0104bda8f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:03.479516 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b08e24ea0874d6f89b62f0104bda8f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:03.486368 2637 tablet_bootstrap.cc:492] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac: Bootstrap starting.
I20260929 16:26:03.488121 2638 tablet_bootstrap.cc:492] T 7b08e24ea0874d6f89b62f0104bda8f0 P 4f645d147fa44b579755a51b53ff0664: Bootstrap starting.
I20260929 16:26:03.489063 2639 tablet_bootstrap.cc:492] T 7b08e24ea0874d6f89b62f0104bda8f0 P 469c940f7a7d40328acb8639ea30596d: Bootstrap starting.
I20260929 16:26:03.489159 2637 tablet_bootstrap.cc:654] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:03.490093 2637 log.cc:824] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:03.490454 2638 tablet_bootstrap.cc:654] T 7b08e24ea0874d6f89b62f0104bda8f0 P 4f645d147fa44b579755a51b53ff0664: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:03.491235 2638 log.cc:824] T 7b08e24ea0874d6f89b62f0104bda8f0 P 4f645d147fa44b579755a51b53ff0664: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:03.491269 2639 tablet_bootstrap.cc:654] T 7b08e24ea0874d6f89b62f0104bda8f0 P 469c940f7a7d40328acb8639ea30596d: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:03.492189 2639 log.cc:824] T 7b08e24ea0874d6f89b62f0104bda8f0 P 469c940f7a7d40328acb8639ea30596d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:03.493309 2637 tablet_bootstrap.cc:492] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac: No bootstrap required, opened a new log
I20260929 16:26:03.493359 2638 tablet_bootstrap.cc:492] T 7b08e24ea0874d6f89b62f0104bda8f0 P 4f645d147fa44b579755a51b53ff0664: No bootstrap required, opened a new log
I20260929 16:26:03.493528 2637 ts_tablet_manager.cc:1397] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 16:26:03.493549 2638 ts_tablet_manager.cc:1397] T 7b08e24ea0874d6f89b62f0104bda8f0 P 4f645d147fa44b579755a51b53ff0664: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:26:03.494280 2639 tablet_bootstrap.cc:492] T 7b08e24ea0874d6f89b62f0104bda8f0 P 469c940f7a7d40328acb8639ea30596d: No bootstrap required, opened a new log
I20260929 16:26:03.494481 2639 ts_tablet_manager.cc:1397] T 7b08e24ea0874d6f89b62f0104bda8f0 P 469c940f7a7d40328acb8639ea30596d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:26:03.497026 2637 raft_consensus.cc:348] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } }
I20260929 16:26:03.497350 2637 raft_consensus.cc:374] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:03.497416 2637 raft_consensus.cc:729] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7e4632bf4d43df9a0d3f0011c522ac, State: Initialized, Role: FOLLOWER
I20260929 16:26:03.497912 2637 consensus_queue.cc:260] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } }
I20260929 16:26:03.497809 2638 raft_consensus.cc:348] T 7b08e24ea0874d6f89b62f0104bda8f0 P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } }
I20260929 16:26:03.498085 2638 raft_consensus.cc:374] T 7b08e24ea0874d6f89b62f0104bda8f0 P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:03.498132 2638 raft_consensus.cc:729] T 7b08e24ea0874d6f89b62f0104bda8f0 P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f645d147fa44b579755a51b53ff0664, State: Initialized, Role: FOLLOWER
I20260929 16:26:03.498606 2638 consensus_queue.cc:260] T 7b08e24ea0874d6f89b62f0104bda8f0 P 4f645d147fa44b579755a51b53ff0664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } }
I20260929 16:26:03.498706 2639 raft_consensus.cc:348] T 7b08e24ea0874d6f89b62f0104bda8f0 P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } }
I20260929 16:26:03.498931 2637 ts_tablet_manager.cc:1428] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 16:26:03.498957 2639 raft_consensus.cc:374] T 7b08e24ea0874d6f89b62f0104bda8f0 P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:03.499025 2639 raft_consensus.cc:729] T 7b08e24ea0874d6f89b62f0104bda8f0 P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469c940f7a7d40328acb8639ea30596d, State: Initialized, Role: FOLLOWER
I20260929 16:26:03.499302 2334 heartbeater.cc:503] Master 127.0.98.62:34265 was elected leader, sending a full tablet report...
I20260929 16:26:03.499464 2639 consensus_queue.cc:260] T 7b08e24ea0874d6f89b62f0104bda8f0 P 469c940f7a7d40328acb8639ea30596d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } }
I20260929 16:26:03.500119 2474 heartbeater.cc:503] Master 127.0.98.62:34265 was elected leader, sending a full tablet report...
I20260929 16:26:03.500522 2639 ts_tablet_manager.cc:1428] T 7b08e24ea0874d6f89b62f0104bda8f0 P 469c940f7a7d40328acb8639ea30596d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:26:03.501094 2638 ts_tablet_manager.cc:1428] T 7b08e24ea0874d6f89b62f0104bda8f0 P 4f645d147fa44b579755a51b53ff0664: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 16:26:03.501098 2614 heartbeater.cc:503] Master 127.0.98.62:34265 was elected leader, sending a full tablet report...
W20260929 16:26:03.666422 2615 tablet.cc:2369] T 7b08e24ea0874d6f89b62f0104bda8f0 P 4f645d147fa44b579755a51b53ff0664: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:26:03.669041 2335 tablet.cc:2369] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:26:03.678802 2475 tablet.cc:2369] T 7b08e24ea0874d6f89b62f0104bda8f0 P 469c940f7a7d40328acb8639ea30596d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:26:03.725685 2643 raft_consensus.cc:482] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:26:03.725942 2643 raft_consensus.cc:504] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } }
I20260929 16:26:03.727269 2643 leader_election.cc:288] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f645d147fa44b579755a51b53ff0664 (127.0.98.3:38465), 469c940f7a7d40328acb8639ea30596d (127.0.98.2:38023)
I20260929 16:26:03.731570 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.727745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54967 (local address 127.0.98.3:38465)
0929 16:26:03.727948 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:03.727954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:03.728061 (+ 107us) server_negotiation.cc:407] Connection header received
0929 16:26:03.728257 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:03.728261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:03.728304 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:03.728388 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:03.728681 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.729352 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.730551 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.730966 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.731094 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.731157 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:03.731341 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 16:26:03.731422 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260929 16:26:03.731600 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.727572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38465 (local address 127.0.98.1:54967)
0929 16:26:03.727947 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:03.727963 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:03.728123 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:03.728400 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:03.728402 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:03.728445 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:03.728563 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.728569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.729467 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.729472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.730401 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.730419 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.731101 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.731104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.731203 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.731226 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:26:03.731326 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 16:26:03.731412 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":32,"thread_start_us":111,"threads_started":1}
I20260929 16:26:03.731891 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.727685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41295 (local address 127.0.98.2:38023)
0929 16:26:03.727852 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:03.727858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:03.728150 (+ 292us) server_negotiation.cc:407] Connection header received
0929 16:26:03.728381 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:03.728384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:03.728439 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:03.728544 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:03.728901 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.729521 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.730718 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.731177 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.731320 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.731388 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:03.731644 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 16:26:03.731745 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:26:03.731911 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.727572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38023 (local address 127.0.98.1:41295)
0929 16:26:03.728023 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:03.728036 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:03.728237 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:03.728554 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:03.728557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:03.728574 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:03.728747 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.728757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.729648 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.729654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.730553 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.730563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.731323 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.731327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.731461 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.731493 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:26:03.731614 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 16:26:03.731736 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":118,"thread_start_us":114,"threads_started":1}
I20260929 16:26:03.732549 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b08e24ea0874d6f89b62f0104bda8f0" candidate_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f645d147fa44b579755a51b53ff0664" is_pre_election: true
I20260929 16:26:03.732599 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b08e24ea0874d6f89b62f0104bda8f0" candidate_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469c940f7a7d40328acb8639ea30596d" is_pre_election: true
I20260929 16:26:03.732900 2428 raft_consensus.cc:2454] T 7b08e24ea0874d6f89b62f0104bda8f0 P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a7e4632bf4d43df9a0d3f0011c522ac in term 0.
I20260929 16:26:03.732903 2568 raft_consensus.cc:2454] T 7b08e24ea0874d6f89b62f0104bda8f0 P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a7e4632bf4d43df9a0d3f0011c522ac in term 0.
I20260929 16:26:03.733603 2220 leader_election.cc:302] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2a7e4632bf4d43df9a0d3f0011c522ac, 469c940f7a7d40328acb8639ea30596d, 4f645d147fa44b579755a51b53ff0664; no voters:
I20260929 16:26:03.733939 2643 raft_consensus.cc:2781] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:26:03.734033 2643 raft_consensus.cc:482] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:26:03.734081 2643 raft_consensus.cc:3037] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:03.735297 2643 raft_consensus.cc:504] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } }
I20260929 16:26:03.735754 2643 leader_election.cc:288] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac [CANDIDATE]: Term 1 election: Requested vote from peers 4f645d147fa44b579755a51b53ff0664 (127.0.98.3:38465), 469c940f7a7d40328acb8639ea30596d (127.0.98.2:38023)
I20260929 16:26:03.736236 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b08e24ea0874d6f89b62f0104bda8f0" candidate_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469c940f7a7d40328acb8639ea30596d"
I20260929 16:26:03.736236 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b08e24ea0874d6f89b62f0104bda8f0" candidate_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f645d147fa44b579755a51b53ff0664"
I20260929 16:26:03.736387 2568 raft_consensus.cc:3037] T 7b08e24ea0874d6f89b62f0104bda8f0 P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:03.736387 2428 raft_consensus.cc:3037] T 7b08e24ea0874d6f89b62f0104bda8f0 P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:03.737864 2428 raft_consensus.cc:2454] T 7b08e24ea0874d6f89b62f0104bda8f0 P 469c940f7a7d40328acb8639ea30596d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a7e4632bf4d43df9a0d3f0011c522ac in term 1.
I20260929 16:26:03.738298 2221 leader_election.cc:302] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a7e4632bf4d43df9a0d3f0011c522ac, 469c940f7a7d40328acb8639ea30596d; no voters:
I20260929 16:26:03.738315 2568 raft_consensus.cc:2454] T 7b08e24ea0874d6f89b62f0104bda8f0 P 4f645d147fa44b579755a51b53ff0664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a7e4632bf4d43df9a0d3f0011c522ac in term 1.
I20260929 16:26:03.738570 2643 raft_consensus.cc:2781] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:03.738832 2643 raft_consensus.cc:686] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac [term 1 LEADER]: Becoming Leader. State: Replica: 2a7e4632bf4d43df9a0d3f0011c522ac, State: Running, Role: LEADER
I20260929 16:26:03.739225 2643 consensus_queue.cc:237] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } }
I20260929 16:26:03.742774 2131 catalog_manager.cc:5495] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a7e4632bf4d43df9a0d3f0011c522ac (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } health_report { overall_health: UNKNOWN } } }
Sep 29 16:26:03 dist-test-slave-vq9v krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:03.796135 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.787262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56752 (local address 127.0.98.62:34265)
0929 16:26:03.787489 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:03.787493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:03.787646 (+ 153us) server_negotiation.cc:407] Connection header received
0929 16:26:03.787811 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:03.787814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:03.787898 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:03.787995 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:03.788662 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.789267 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.790079 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.790291 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.790409 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.792806 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:03.792809 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:03.792811 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:03.792829 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:03.794365 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:03.794864 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:03.794870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:03.794872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:03.794944 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:03.795236 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:03.795240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:03.795241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:03.795412 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:03.795514 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:03.795815 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 16:26:03.795982 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":121}
I20260929 16:26:03.796135 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.787177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34265 (local address 127.0.0.1:56752)
0929 16:26:03.787553 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:03.787565 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:03.787710 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:03.788017 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:03.788020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:03.788452 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:03.788540 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.788546 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.789398 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.789402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.789957 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.789964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.790413 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.790416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.790509 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.791130 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:03.791142 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:03.792651 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:03.794505 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:03.794515 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:03.794523 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:03.794754 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:03.795039 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:03.795043 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:03.795045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:03.795124 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:03.795526 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:03.795535 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:03.795643 (+ 108us) client_negotiation.cc:769] Sending connection context
0929 16:26:03.795766 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 16:26:03.795932 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":1,"thread_start_us":134,"threads_started":1}
I20260929 16:26:03.800402 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56752:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 16:26:03.801071 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:26:03.806229 2408 tablet_service.cc:1467] Processing CreateTablet for tablet 474c192c5c844de6aa29d32797c9410a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bee85a38a3c147f4afe807ff09dad43f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:26:03.806178 2268 tablet_service.cc:1467] Processing CreateTablet for tablet 474c192c5c844de6aa29d32797c9410a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bee85a38a3c147f4afe807ff09dad43f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:26:03.806329 2541 tablet_service.cc:1467] Processing CreateTablet for tablet 474c192c5c844de6aa29d32797c9410a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bee85a38a3c147f4afe807ff09dad43f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:26:03.806908 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474c192c5c844de6aa29d32797c9410a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:03.806910 2408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474c192c5c844de6aa29d32797c9410a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:03.806908 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474c192c5c844de6aa29d32797c9410a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:03.809334 2639 tablet_bootstrap.cc:492] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d: Bootstrap starting.
I20260929 16:26:03.809774 2637 tablet_bootstrap.cc:492] T 474c192c5c844de6aa29d32797c9410a P 2a7e4632bf4d43df9a0d3f0011c522ac: Bootstrap starting.
I20260929 16:26:03.810376 2639 tablet_bootstrap.cc:654] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:03.810568 2638 tablet_bootstrap.cc:492] T 474c192c5c844de6aa29d32797c9410a P 4f645d147fa44b579755a51b53ff0664: Bootstrap starting.
I20260929 16:26:03.811134 2637 tablet_bootstrap.cc:654] T 474c192c5c844de6aa29d32797c9410a P 2a7e4632bf4d43df9a0d3f0011c522ac: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:03.811440 2638 tablet_bootstrap.cc:654] T 474c192c5c844de6aa29d32797c9410a P 4f645d147fa44b579755a51b53ff0664: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:03.811774 2639 tablet_bootstrap.cc:492] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d: No bootstrap required, opened a new log
I20260929 16:26:03.811869 2639 ts_tablet_manager.cc:1397] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:26:03.812417 2639 raft_consensus.cc:348] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:03.812521 2637 tablet_bootstrap.cc:492] T 474c192c5c844de6aa29d32797c9410a P 2a7e4632bf4d43df9a0d3f0011c522ac: No bootstrap required, opened a new log
I20260929 16:26:03.812538 2639 raft_consensus.cc:374] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:03.812572 2639 raft_consensus.cc:729] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469c940f7a7d40328acb8639ea30596d, State: Initialized, Role: FOLLOWER
I20260929 16:26:03.812599 2637 ts_tablet_manager.cc:1397] T 474c192c5c844de6aa29d32797c9410a P 2a7e4632bf4d43df9a0d3f0011c522ac: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:26:03.812713 2639 consensus_queue.cc:260] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:03.812784 2638 tablet_bootstrap.cc:492] T 474c192c5c844de6aa29d32797c9410a P 4f645d147fa44b579755a51b53ff0664: No bootstrap required, opened a new log
I20260929 16:26:03.812861 2638 ts_tablet_manager.cc:1397] T 474c192c5c844de6aa29d32797c9410a P 4f645d147fa44b579755a51b53ff0664: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:26:03.812983 2639 ts_tablet_manager.cc:1428] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 16:26:03.813078 2637 raft_consensus.cc:348] T 474c192c5c844de6aa29d32797c9410a P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:03.813344 2638 raft_consensus.cc:348] T 474c192c5c844de6aa29d32797c9410a P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:03.813462 2638 raft_consensus.cc:374] T 474c192c5c844de6aa29d32797c9410a P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:03.813499 2638 raft_consensus.cc:729] T 474c192c5c844de6aa29d32797c9410a P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f645d147fa44b579755a51b53ff0664, State: Initialized, Role: FOLLOWER
I20260929 16:26:03.813632 2638 consensus_queue.cc:260] T 474c192c5c844de6aa29d32797c9410a P 4f645d147fa44b579755a51b53ff0664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:03.813621 2645 tablet_replica.cc:439] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } } }
I20260929 16:26:03.813759 2645 tablet_replica.cc:442] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:26:03.813817 2637 raft_consensus.cc:374] T 474c192c5c844de6aa29d32797c9410a P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:03.813889 2637 raft_consensus.cc:729] T 474c192c5c844de6aa29d32797c9410a P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7e4632bf4d43df9a0d3f0011c522ac, State: Initialized, Role: FOLLOWER
I20260929 16:26:03.813894 2638 ts_tablet_manager.cc:1428] T 474c192c5c844de6aa29d32797c9410a P 4f645d147fa44b579755a51b53ff0664: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:26:03.814026 2637 consensus_queue.cc:260] T 474c192c5c844de6aa29d32797c9410a P 2a7e4632bf4d43df9a0d3f0011c522ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:03.814283 2644 tablet_replica.cc:439] T 474c192c5c844de6aa29d32797c9410a P 4f645d147fa44b579755a51b53ff0664: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } } }
I20260929 16:26:03.814302 2643 tablet_replica.cc:439] T 474c192c5c844de6aa29d32797c9410a P 2a7e4632bf4d43df9a0d3f0011c522ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } } }
I20260929 16:26:03.814427 2643 tablet_replica.cc:442] T 474c192c5c844de6aa29d32797c9410a P 2a7e4632bf4d43df9a0d3f0011c522ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:26:03.814429 2644 tablet_replica.cc:442] T 474c192c5c844de6aa29d32797c9410a P 4f645d147fa44b579755a51b53ff0664: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:26:03.815464 2637 ts_tablet_manager.cc:1428] T 474c192c5c844de6aa29d32797c9410a P 2a7e4632bf4d43df9a0d3f0011c522ac: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260929 16:26:03.854393 2645 raft_consensus.cc:482] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:26:03.854543 2645 raft_consensus.cc:504] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:03.855656 2645 leader_election.cc:288] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a7e4632bf4d43df9a0d3f0011c522ac (127.0.98.1:44297), 4f645d147fa44b579755a51b53ff0664 (127.0.98.3:38465)
I20260929 16:26:03.860496 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.855974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47333 (local address 127.0.98.1:44297)
0929 16:26:03.856116 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:03.856120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:03.856405 (+ 285us) server_negotiation.cc:407] Connection header received
0929 16:26:03.856534 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:03.856537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:03.856593 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:03.856679 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:03.857104 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.857877 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.859185 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.859816 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.859904 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.859961 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:03.860233 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 16:26:03.860318 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:26:03.860509 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.855919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44297 (local address 127.0.98.2:47333)
0929 16:26:03.856282 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:03.856297 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:03.856424 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:03.856800 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:03.856802 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:03.856817 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:03.856966 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.856975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.858000 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.858003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.859050 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.859063 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.859963 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.859967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.860078 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.860104 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 16:26:03.860193 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 16:26:03.860270 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":102,"threads_started":1}
I20260929 16:26:03.861056 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.856007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40271 (local address 127.0.98.3:38465)
0929 16:26:03.856312 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:03.856316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:03.856554 (+ 238us) server_negotiation.cc:407] Connection header received
0929 16:26:03.856781 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:03.856784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:03.856831 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:03.856914 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:03.857690 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.858397 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.859758 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.860376 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.860473 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.860538 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:03.860853 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 16:26:03.860934 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260929 16:26:03.861101 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.855972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38465 (local address 127.0.98.2:40271)
0929 16:26:03.856411 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:03.856425 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:03.856556 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:03.856945 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:03.856948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:03.856964 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:03.857553 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.857564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.858541 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.858543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.859613 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.859623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.860599 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.860606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.860714 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.860747 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:26:03.860820 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 16:26:03.860948 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":79,"threads_started":1}
I20260929 16:26:03.861485 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474c192c5c844de6aa29d32797c9410a" candidate_uuid: "469c940f7a7d40328acb8639ea30596d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" is_pre_election: true
I20260929 16:26:03.861631 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474c192c5c844de6aa29d32797c9410a" candidate_uuid: "469c940f7a7d40328acb8639ea30596d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f645d147fa44b579755a51b53ff0664" is_pre_election: true
I20260929 16:26:03.861752 2288 raft_consensus.cc:2454] T 474c192c5c844de6aa29d32797c9410a P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 469c940f7a7d40328acb8639ea30596d in term 0.
I20260929 16:26:03.861793 2568 raft_consensus.cc:2454] T 474c192c5c844de6aa29d32797c9410a P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 469c940f7a7d40328acb8639ea30596d in term 0.
I20260929 16:26:03.862208 2360 leader_election.cc:302] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2a7e4632bf4d43df9a0d3f0011c522ac, 469c940f7a7d40328acb8639ea30596d, 4f645d147fa44b579755a51b53ff0664; no voters:
I20260929 16:26:03.862540 2645 raft_consensus.cc:2781] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:26:03.862625 2645 raft_consensus.cc:482] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:26:03.862681 2645 raft_consensus.cc:3037] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:03.863750 2645 raft_consensus.cc:504] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:03.864152 2645 leader_election.cc:288] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d [CANDIDATE]: Term 1 election: Requested vote from peers 2a7e4632bf4d43df9a0d3f0011c522ac (127.0.98.1:44297), 4f645d147fa44b579755a51b53ff0664 (127.0.98.3:38465)
I20260929 16:26:03.864570 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474c192c5c844de6aa29d32797c9410a" candidate_uuid: "469c940f7a7d40328acb8639ea30596d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac"
I20260929 16:26:03.864589 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474c192c5c844de6aa29d32797c9410a" candidate_uuid: "469c940f7a7d40328acb8639ea30596d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f645d147fa44b579755a51b53ff0664"
I20260929 16:26:03.864714 2288 raft_consensus.cc:3037] T 474c192c5c844de6aa29d32797c9410a P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:03.864749 2568 raft_consensus.cc:3037] T 474c192c5c844de6aa29d32797c9410a P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:03.865751 2568 raft_consensus.cc:2454] T 474c192c5c844de6aa29d32797c9410a P 4f645d147fa44b579755a51b53ff0664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 469c940f7a7d40328acb8639ea30596d in term 1.
I20260929 16:26:03.865765 2288 raft_consensus.cc:2454] T 474c192c5c844de6aa29d32797c9410a P 2a7e4632bf4d43df9a0d3f0011c522ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 469c940f7a7d40328acb8639ea30596d in term 1.
I20260929 16:26:03.866068 2360 leader_election.cc:302] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 469c940f7a7d40328acb8639ea30596d, 4f645d147fa44b579755a51b53ff0664; no voters:
I20260929 16:26:03.866250 2645 raft_consensus.cc:2781] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:03.866444 2645 raft_consensus.cc:686] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d [term 1 LEADER]: Becoming Leader. State: Replica: 469c940f7a7d40328acb8639ea30596d, State: Running, Role: LEADER
I20260929 16:26:03.866842 2645 consensus_queue.cc:237] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:03.869258 2661 tablet_replica.cc:439] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d: TxnStatusTablet state changed. Reason: New leader 469c940f7a7d40328acb8639ea30596d. Latest consensus state: current_term: 1 leader_uuid: "469c940f7a7d40328acb8639ea30596d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } } }
I20260929 16:26:03.869422 2661 tablet_replica.cc:442] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:26:03.869717 2663 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 16:26:03.869951 2663 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 16:26:03.870540 2129 catalog_manager.cc:5495] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d reported cstate change: term changed from 0 to 1, leader changed from <none> to 469c940f7a7d40328acb8639ea30596d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "469c940f7a7d40328acb8639ea30596d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } health_report { overall_health: UNKNOWN } } }
I20260929 16:26:03.885180 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.880334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38023 (local address 127.0.0.1:58678)
0929 16:26:03.880597 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:03.880613 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:03.880736 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:03.881244 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:03.881247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:03.881256 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:26:03.881373 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.881380 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.882246 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.882250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.883241 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.883254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.883785 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.883789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.883907 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.883977 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:26:03.884795 (+ 818us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:26:03.884816 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:26:03.884902 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 16:26:03.885013 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260929 16:26:03.885262 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.880453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58678 (local address 127.0.98.2:38023)
0929 16:26:03.880567 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:03.880572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:03.880661 (+ 89us) server_negotiation.cc:407] Connection header received
0929 16:26:03.880942 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:03.880945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:03.881045 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:03.881192 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:03.881597 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.882135 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.883384 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.883614 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.883781 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.884159 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:03.884695 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:03.884816 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:03.885005 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 16:26:03.885078 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 16:26:03.891171 2568 raft_consensus.cc:1261] T 474c192c5c844de6aa29d32797c9410a P 4f645d147fa44b579755a51b53ff0664 [term 1 FOLLOWER]: Refusing update from remote peer 469c940f7a7d40328acb8639ea30596d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:03.892088 2288 raft_consensus.cc:1261] T 474c192c5c844de6aa29d32797c9410a P 2a7e4632bf4d43df9a0d3f0011c522ac [term 1 FOLLOWER]: Refusing update from remote peer 469c940f7a7d40328acb8639ea30596d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:03.892023 2661 consensus_queue.cc:1035] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:03.892666 2661 consensus_queue.cc:1035] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:03.895853 2643 tablet_replica.cc:439] T 474c192c5c844de6aa29d32797c9410a P 2a7e4632bf4d43df9a0d3f0011c522ac: TxnStatusTablet state changed. Reason: New leader 469c940f7a7d40328acb8639ea30596d. Latest consensus state: current_term: 1 leader_uuid: "469c940f7a7d40328acb8639ea30596d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } } }
I20260929 16:26:03.896008 2643 tablet_replica.cc:442] T 474c192c5c844de6aa29d32797c9410a P 2a7e4632bf4d43df9a0d3f0011c522ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:26:03.896744 2644 tablet_replica.cc:439] T 474c192c5c844de6aa29d32797c9410a P 4f645d147fa44b579755a51b53ff0664: TxnStatusTablet state changed. Reason: New leader 469c940f7a7d40328acb8639ea30596d. Latest consensus state: current_term: 1 leader_uuid: "469c940f7a7d40328acb8639ea30596d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } } }
I20260929 16:26:03.896888 2644 tablet_replica.cc:442] T 474c192c5c844de6aa29d32797c9410a P 4f645d147fa44b579755a51b53ff0664: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:26:03.901325 2643 tablet_replica.cc:439] T 474c192c5c844de6aa29d32797c9410a P 2a7e4632bf4d43df9a0d3f0011c522ac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "469c940f7a7d40328acb8639ea30596d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } } }
I20260929 16:26:03.901496 2643 tablet_replica.cc:442] T 474c192c5c844de6aa29d32797c9410a P 2a7e4632bf4d43df9a0d3f0011c522ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:26:03.902015 2644 tablet_replica.cc:439] T 474c192c5c844de6aa29d32797c9410a P 4f645d147fa44b579755a51b53ff0664: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "469c940f7a7d40328acb8639ea30596d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } } }
I20260929 16:26:03.902143 2644 tablet_replica.cc:442] T 474c192c5c844de6aa29d32797c9410a P 4f645d147fa44b579755a51b53ff0664: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:26:03.902163 2661 tablet_replica.cc:439] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "469c940f7a7d40328acb8639ea30596d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } } }
I20260929 16:26:03.902285 2661 tablet_replica.cc:442] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:26:03.902400 2668 mvcc.cc:205] Tried to move back new op lower bound from 7334703775289942016 to 7334703775201710080. Current Snapshot: MvccSnapshot[applied={T|T < 7334703775289942016}]
I20260929 16:26:03.902571 2667 mvcc.cc:205] Tried to move back new op lower bound from 7334703775289942016 to 7334703775201710080. Current Snapshot: MvccSnapshot[applied={T|T < 7334703775289942016}]
I20260929 16:26:03.905051 2671 tablet_replica.cc:439] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "469c940f7a7d40328acb8639ea30596d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } } }
I20260929 16:26:03.905248 2671 tablet_replica.cc:442] T 474c192c5c844de6aa29d32797c9410a P 469c940f7a7d40328acb8639ea30596d: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:26:03.914494 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.910822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39062 (local address 127.0.98.1:44297)
0929 16:26:03.910972 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:03.910979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:03.910997 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:26:03.911090 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:03.911093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:03.911146 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:03.911276 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:03.911535 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.912063 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.912933 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.913166 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.913330 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.913615 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:03.914083 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:03.914215 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:03.914277 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 16:26:03.914337 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:26:03.927614 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.923397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34837 (local address 127.0.98.2:38023)
0929 16:26:03.923756 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:03.923763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:03.923782 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:26:03.923986 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:03.923990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:03.924048 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:03.924146 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:03.924445 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.925030 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.926242 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.926507 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.926637 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.927023 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:03.927167 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:03.927278 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:03.927393 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 16:26:03.927450 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264}
I20260929 16:26:03.927683 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.923307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38023 (local address 127.0.98.1:34837)
0929 16:26:03.923697 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:03.923720 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:03.923842 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:03.924161 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:03.924165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:03.924174 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:26:03.924323 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.924329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.925164 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.925168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.926059 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.926072 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.926642 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.926646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.926761 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.926810 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:26:03.927294 (+ 484us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:26:03.927304 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 16:26:03.927378 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 16:26:03.927469 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":158,"threads_started":1}
I20260929 16:26:03.935951 2568 raft_consensus.cc:1261] T 7b08e24ea0874d6f89b62f0104bda8f0 P 4f645d147fa44b579755a51b53ff0664 [term 1 FOLLOWER]: Refusing update from remote peer 2a7e4632bf4d43df9a0d3f0011c522ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:03.936064 2428 raft_consensus.cc:1261] T 7b08e24ea0874d6f89b62f0104bda8f0 P 469c940f7a7d40328acb8639ea30596d [term 1 FOLLOWER]: Refusing update from remote peer 2a7e4632bf4d43df9a0d3f0011c522ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:03.936460 2648 consensus_queue.cc:1035] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:03.936630 2643 consensus_queue.cc:1035] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:03.960943 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.956793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50277 (local address 127.0.98.1:44297)
0929 16:26:03.956981 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:03.956985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:03.957130 (+ 145us) server_negotiation.cc:407] Connection header received
0929 16:26:03.957315 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:03.957318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:03.957388 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:03.957468 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:03.957883 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.958484 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.959445 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.959756 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.959854 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.960267 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:03.960376 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:03.960481 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:03.960730 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 16:26:03.960785 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260929 16:26:03.961313 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:03.956644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44297 (local address 127.0.98.2:50277)
0929 16:26:03.957020 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:03.957038 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:03.957163 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:03.957575 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:03.957577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:03.957612 (+ 35us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:26:03.957742 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.957748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.958606 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.958609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.959341 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:03.959353 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:03.959932 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:03.959943 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:03.960051 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:03.960109 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:26:03.960615 (+ 506us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:26:03.960621 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:26:03.960713 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 16:26:03.960795 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":140,"threads_started":1}
I20260929 16:26:03.987854 2130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:03.988075 2130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:03.991201 2130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a: Sending DeleteTablet for 3 replicas of tablet 7b08e24ea0874d6f89b62f0104bda8f0
I20260929 16:26:03.992313 2268 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b08e24ea0874d6f89b62f0104bda8f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39060
I20260929 16:26:03.992810 2688 tablet_replica.cc:330] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac: stopping tablet replica
I20260929 16:26:03.993072 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b08e24ea0874d6f89b62f0104bda8f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35410
I20260929 16:26:03.993299 2688 raft_consensus.cc:2229] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:26:03.993690 2688 raft_consensus.cc:2258] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:03.994185 2408 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b08e24ea0874d6f89b62f0104bda8f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58670
I20260929 16:26:03.994448 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 16:26:03.994880 2689 tablet_replica.cc:330] T 7b08e24ea0874d6f89b62f0104bda8f0 P 4f645d147fa44b579755a51b53ff0664: stopping tablet replica
W20260929 16:26:03.994901 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:26:03.995280 2689 raft_consensus.cc:2229] T 7b08e24ea0874d6f89b62f0104bda8f0 P 4f645d147fa44b579755a51b53ff0664 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:03.995597 2689 raft_consensus.cc:2258] T 7b08e24ea0874d6f89b62f0104bda8f0 P 4f645d147fa44b579755a51b53ff0664 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:03.995970 2690 tablet_replica.cc:330] T 7b08e24ea0874d6f89b62f0104bda8f0 P 469c940f7a7d40328acb8639ea30596d: stopping tablet replica
I20260929 16:26:03.997344 2689 ts_tablet_manager.cc:1905] T 7b08e24ea0874d6f89b62f0104bda8f0 P 4f645d147fa44b579755a51b53ff0664: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:03.997716 2690 raft_consensus.cc:2229] T 7b08e24ea0874d6f89b62f0104bda8f0 P 469c940f7a7d40328acb8639ea30596d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:03.997964 2690 raft_consensus.cc:2258] T 7b08e24ea0874d6f89b62f0104bda8f0 P 469c940f7a7d40328acb8639ea30596d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:03.999630 2690 ts_tablet_manager.cc:1905] T 7b08e24ea0874d6f89b62f0104bda8f0 P 469c940f7a7d40328acb8639ea30596d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:04.001411 2268 tablet_service.cc:1467] Processing CreateTablet for tablet dfac9a92ad764f6f8b113b1f500e058f (DEFAULT_TABLE table=test-table [id=afa6dcd5725145778e4cd0ca0367454e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:04.001711 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfac9a92ad764f6f8b113b1f500e058f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:04.004189 2637 tablet_bootstrap.cc:492] T dfac9a92ad764f6f8b113b1f500e058f P 2a7e4632bf4d43df9a0d3f0011c522ac: Bootstrap starting.
I20260929 16:26:04.004360 2408 tablet_service.cc:1467] Processing CreateTablet for tablet dfac9a92ad764f6f8b113b1f500e058f (DEFAULT_TABLE table=test-table [id=afa6dcd5725145778e4cd0ca0367454e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:04.004622 2408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfac9a92ad764f6f8b113b1f500e058f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:04.005411 2637 tablet_bootstrap.cc:654] T dfac9a92ad764f6f8b113b1f500e058f P 2a7e4632bf4d43df9a0d3f0011c522ac: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:04.006721 2689 ts_tablet_manager.cc:1918] T 7b08e24ea0874d6f89b62f0104bda8f0 P 4f645d147fa44b579755a51b53ff0664: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:26:04.007141 2690 ts_tablet_manager.cc:1918] T 7b08e24ea0874d6f89b62f0104bda8f0 P 469c940f7a7d40328acb8639ea30596d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:26:04.007019 2541 tablet_service.cc:1467] Processing CreateTablet for tablet dfac9a92ad764f6f8b113b1f500e058f (DEFAULT_TABLE table=test-table [id=afa6dcd5725145778e4cd0ca0367454e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:04.007663 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfac9a92ad764f6f8b113b1f500e058f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:04.009382 2688 ts_tablet_manager.cc:1905] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:04.010520 2638 tablet_bootstrap.cc:492] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664: Bootstrap starting.
I20260929 16:26:04.011507 2638 tablet_bootstrap.cc:654] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:04.012321 2688 ts_tablet_manager.cc:1918] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:26:04.012408 2688 log.cc:1196] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-0/wal/wals/7b08e24ea0874d6f89b62f0104bda8f0
I20260929 16:26:04.012727 2688 ts_tablet_manager.cc:1939] T 7b08e24ea0874d6f89b62f0104bda8f0 P 2a7e4632bf4d43df9a0d3f0011c522ac: Deleting consensus metadata
I20260929 16:26:04.013686 2689 log.cc:1196] T 7b08e24ea0874d6f89b62f0104bda8f0 P 4f645d147fa44b579755a51b53ff0664: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/7b08e24ea0874d6f89b62f0104bda8f0
I20260929 16:26:04.007215 2639 tablet_bootstrap.cc:492] T dfac9a92ad764f6f8b113b1f500e058f P 469c940f7a7d40328acb8639ea30596d: Bootstrap starting.
I20260929 16:26:04.013976 2637 tablet_bootstrap.cc:492] T dfac9a92ad764f6f8b113b1f500e058f P 2a7e4632bf4d43df9a0d3f0011c522ac: No bootstrap required, opened a new log
I20260929 16:26:04.013980 2115 catalog_manager.cc:4841] TS 2a7e4632bf4d43df9a0d3f0011c522ac (127.0.98.1:44297): tablet 7b08e24ea0874d6f89b62f0104bda8f0 (table test-table [id=89b0af9609bf4f19a9cd46c60e5f07c0]) successfully deleted
I20260929 16:26:04.014056 2637 ts_tablet_manager.cc:1397] T dfac9a92ad764f6f8b113b1f500e058f P 2a7e4632bf4d43df9a0d3f0011c522ac: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260929 16:26:04.014530 2637 raft_consensus.cc:348] T dfac9a92ad764f6f8b113b1f500e058f P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:04.014649 2637 raft_consensus.cc:374] T dfac9a92ad764f6f8b113b1f500e058f P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:04.014683 2637 raft_consensus.cc:729] T dfac9a92ad764f6f8b113b1f500e058f P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7e4632bf4d43df9a0d3f0011c522ac, State: Initialized, Role: FOLLOWER
I20260929 16:26:04.014879 2639 tablet_bootstrap.cc:654] T dfac9a92ad764f6f8b113b1f500e058f P 469c940f7a7d40328acb8639ea30596d: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:04.014838 2637 consensus_queue.cc:260] T dfac9a92ad764f6f8b113b1f500e058f P 2a7e4632bf4d43df9a0d3f0011c522ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:04.015156 2637 ts_tablet_manager.cc:1428] T dfac9a92ad764f6f8b113b1f500e058f P 2a7e4632bf4d43df9a0d3f0011c522ac: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:26:04.016949 2690 log.cc:1196] T 7b08e24ea0874d6f89b62f0104bda8f0 P 469c940f7a7d40328acb8639ea30596d: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/7b08e24ea0874d6f89b62f0104bda8f0
I20260929 16:26:04.017547 2689 ts_tablet_manager.cc:1939] T 7b08e24ea0874d6f89b62f0104bda8f0 P 4f645d147fa44b579755a51b53ff0664: Deleting consensus metadata
I20260929 16:26:04.018007 2690 ts_tablet_manager.cc:1939] T 7b08e24ea0874d6f89b62f0104bda8f0 P 469c940f7a7d40328acb8639ea30596d: Deleting consensus metadata
I20260929 16:26:04.019621 2116 catalog_manager.cc:4841] TS 4f645d147fa44b579755a51b53ff0664 (127.0.98.3:38465): tablet 7b08e24ea0874d6f89b62f0104bda8f0 (table test-table [id=89b0af9609bf4f19a9cd46c60e5f07c0]) successfully deleted
I20260929 16:26:04.021281 2117 catalog_manager.cc:4841] TS 469c940f7a7d40328acb8639ea30596d (127.0.98.2:38023): tablet 7b08e24ea0874d6f89b62f0104bda8f0 (table test-table [id=89b0af9609bf4f19a9cd46c60e5f07c0]) successfully deleted
I20260929 16:26:04.022046 2639 tablet_bootstrap.cc:492] T dfac9a92ad764f6f8b113b1f500e058f P 469c940f7a7d40328acb8639ea30596d: No bootstrap required, opened a new log
I20260929 16:26:04.022110 2638 tablet_bootstrap.cc:492] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664: No bootstrap required, opened a new log
I20260929 16:26:04.022145 2639 ts_tablet_manager.cc:1397] T dfac9a92ad764f6f8b113b1f500e058f P 469c940f7a7d40328acb8639ea30596d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260929 16:26:04.022202 2638 ts_tablet_manager.cc:1397] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260929 16:26:04.022636 2639 raft_consensus.cc:348] T dfac9a92ad764f6f8b113b1f500e058f P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:04.022758 2639 raft_consensus.cc:374] T dfac9a92ad764f6f8b113b1f500e058f P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:04.022787 2639 raft_consensus.cc:729] T dfac9a92ad764f6f8b113b1f500e058f P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469c940f7a7d40328acb8639ea30596d, State: Initialized, Role: FOLLOWER
I20260929 16:26:04.022737 2638 raft_consensus.cc:348] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:04.022864 2638 raft_consensus.cc:374] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:04.022902 2638 raft_consensus.cc:729] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f645d147fa44b579755a51b53ff0664, State: Initialized, Role: FOLLOWER
I20260929 16:26:04.022919 2639 consensus_queue.cc:260] T dfac9a92ad764f6f8b113b1f500e058f P 469c940f7a7d40328acb8639ea30596d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:04.023036 2638 consensus_queue.cc:260] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:04.023182 2639 ts_tablet_manager.cc:1428] T dfac9a92ad764f6f8b113b1f500e058f P 469c940f7a7d40328acb8639ea30596d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:26:04.023335 2638 ts_tablet_manager.cc:1428] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:26:04.031735 2644 raft_consensus.cc:482] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:26:04.031885 2644 raft_consensus.cc:504] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:04.033257 2644 leader_election.cc:288] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a7e4632bf4d43df9a0d3f0011c522ac (127.0.98.1:44297), 469c940f7a7d40328acb8639ea30596d (127.0.98.2:38023)
I20260929 16:26:04.038744 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:04.033842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59029 (local address 127.0.98.2:38023)
0929 16:26:04.033964 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:04.033969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:04.034538 (+ 569us) server_negotiation.cc:407] Connection header received
0929 16:26:04.034711 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:04.034713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:04.034758 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:04.034823 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:04.035238 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:04.035936 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:04.037405 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:04.038020 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:04.038102 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:04.038170 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:04.038523 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 16:26:04.038605 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 16:26:04.039194 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:04.033591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38023 (local address 127.0.98.3:59029)
0929 16:26:04.034424 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:04.034437 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:04.034599 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:04.035018 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:04.035020 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:04.035037 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:04.035117 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:04.035122 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:04.036066 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:04.036069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:04.037264 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:04.037275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:04.038269 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:04.038272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:04.038375 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:04.038399 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:26:04.038942 (+ 543us) client_negotiation.cc:240] Negotiation successful
0929 16:26:04.039049 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":2,"thread_start_us":47,"threads_started":1}
I20260929 16:26:04.039603 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfac9a92ad764f6f8b113b1f500e058f" candidate_uuid: "4f645d147fa44b579755a51b53ff0664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469c940f7a7d40328acb8639ea30596d" is_pre_election: true
I20260929 16:26:04.039763 2428 raft_consensus.cc:2454] T dfac9a92ad764f6f8b113b1f500e058f P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f645d147fa44b579755a51b53ff0664 in term 0.
I20260929 16:26:04.040166 2501 leader_election.cc:302] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 469c940f7a7d40328acb8639ea30596d, 4f645d147fa44b579755a51b53ff0664; no voters:
I20260929 16:26:04.040395 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:04.033876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46885 (local address 127.0.98.1:44297)
0929 16:26:04.033990 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:04.033994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:04.034206 (+ 212us) server_negotiation.cc:407] Connection header received
0929 16:26:04.034361 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:04.034365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:04.034415 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:04.034486 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:04.034932 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:04.035757 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:04.037406 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:04.038022 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:04.038102 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:04.038170 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:04.040200 (+ 2030us) server_negotiation.cc:299] Negotiation successful
0929 16:26:04.040287 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:26:04.040457 2644 raft_consensus.cc:2781] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:26:04.040503 2644 raft_consensus.cc:482] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:26:04.040547 2644 raft_consensus.cc:3037] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:04.040992 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:04.033415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44297 (local address 127.0.98.3:46885)
0929 16:26:04.034118 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:04.034129 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:04.034252 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:04.034631 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:04.034637 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:04.034653 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:04.034763 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:04.034769 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:04.035884 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:04.035887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:04.037264 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:04.037275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:04.039946 (+ 2671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:04.039953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:04.040056 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:04.040078 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:26:04.040777 (+ 699us) client_negotiation.cc:240] Negotiation successful
0929 16:26:04.040863 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":29,"thread_start_us":118,"threads_started":1}
I20260929 16:26:04.041591 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfac9a92ad764f6f8b113b1f500e058f" candidate_uuid: "4f645d147fa44b579755a51b53ff0664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" is_pre_election: true
I20260929 16:26:04.041741 2288 raft_consensus.cc:2454] T dfac9a92ad764f6f8b113b1f500e058f P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f645d147fa44b579755a51b53ff0664 in term 0.
I20260929 16:26:04.041952 2644 raft_consensus.cc:504] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:04.042315 2644 leader_election.cc:288] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664 [CANDIDATE]: Term 1 election: Requested vote from peers 2a7e4632bf4d43df9a0d3f0011c522ac (127.0.98.1:44297), 469c940f7a7d40328acb8639ea30596d (127.0.98.2:38023)
I20260929 16:26:04.042645 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfac9a92ad764f6f8b113b1f500e058f" candidate_uuid: "4f645d147fa44b579755a51b53ff0664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac"
I20260929 16:26:04.042689 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfac9a92ad764f6f8b113b1f500e058f" candidate_uuid: "4f645d147fa44b579755a51b53ff0664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469c940f7a7d40328acb8639ea30596d"
I20260929 16:26:04.042771 2288 raft_consensus.cc:3037] T dfac9a92ad764f6f8b113b1f500e058f P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:04.042794 2428 raft_consensus.cc:3037] T dfac9a92ad764f6f8b113b1f500e058f P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:04.043888 2288 raft_consensus.cc:2454] T dfac9a92ad764f6f8b113b1f500e058f P 2a7e4632bf4d43df9a0d3f0011c522ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f645d147fa44b579755a51b53ff0664 in term 1.
I20260929 16:26:04.043903 2428 raft_consensus.cc:2454] T dfac9a92ad764f6f8b113b1f500e058f P 469c940f7a7d40328acb8639ea30596d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f645d147fa44b579755a51b53ff0664 in term 1.
I20260929 16:26:04.044225 2501 leader_election.cc:302] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 469c940f7a7d40328acb8639ea30596d, 4f645d147fa44b579755a51b53ff0664; no voters:
I20260929 16:26:04.044432 2644 raft_consensus.cc:2781] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:04.044687 2644 raft_consensus.cc:686] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664 [term 1 LEADER]: Becoming Leader. State: Replica: 4f645d147fa44b579755a51b53ff0664, State: Running, Role: LEADER
I20260929 16:26:04.044955 2644 consensus_queue.cc:237] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:04.047363 2131 catalog_manager.cc:5495] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f645d147fa44b579755a51b53ff0664 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4f645d147fa44b579755a51b53ff0664" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } health_report { overall_health: HEALTHY } } }
I20260929 16:26:04.070822 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:04.066777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35418 (local address 127.0.98.3:38465)
0929 16:26:04.067188 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:04.067192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:04.067203 (+ 11us) server_negotiation.cc:407] Connection header received
0929 16:26:04.067273 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:04.067275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:04.067327 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:04.067448 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:04.067882 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:04.068394 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:04.069260 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:04.069504 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:04.069645 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:04.069924 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:04.070398 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:04.070544 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:04.070607 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 16:26:04.070669 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332}
I20260929 16:26:04.083164 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:04.078459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58139 (local address 127.0.98.2:38023)
0929 16:26:04.078599 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:04.078604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:04.078897 (+ 293us) server_negotiation.cc:407] Connection header received
0929 16:26:04.079067 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:04.079071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:04.079151 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:04.079228 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:04.079579 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:04.080169 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:04.081825 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:04.082207 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:04.082314 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:04.082665 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:04.082757 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:04.082838 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:04.082936 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 16:26:04.082993 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:26:04.083186 2700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:04.078411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38023 (local address 127.0.98.3:58139)
0929 16:26:04.078791 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:04.078804 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:04.078946 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:04.079238 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:04.079242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:04.079249 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:26:04.079458 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:04.079466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:04.080308 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:04.080312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:04.081693 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:04.081704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:04.082330 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:04.082334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:04.082437 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:04.082505 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:26:04.082853 (+ 348us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:26:04.082862 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:26:04.082925 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 16:26:04.083029 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":126,"threads_started":1}
I20260929 16:26:04.091660 2428 raft_consensus.cc:1261] T dfac9a92ad764f6f8b113b1f500e058f P 469c940f7a7d40328acb8639ea30596d [term 1 FOLLOWER]: Refusing update from remote peer 4f645d147fa44b579755a51b53ff0664: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:04.092344 2288 raft_consensus.cc:1261] T dfac9a92ad764f6f8b113b1f500e058f P 2a7e4632bf4d43df9a0d3f0011c522ac [term 1 FOLLOWER]: Refusing update from remote peer 4f645d147fa44b579755a51b53ff0664: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:04.092336 2696 consensus_queue.cc:1035] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:04.092731 2696 consensus_queue.cc:1035] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:04.118939 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:04.114421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38465 (local address 127.0.98.2:43965)
0929 16:26:04.114607 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:04.114622 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:04.114711 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:04.115141 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:04.115144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:04.115154 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:26:04.115263 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:04.115270 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:04.116527 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:04.116530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:04.117404 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:04.117420 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:04.118046 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:04.118051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:04.118156 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:04.118196 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:26:04.118567 (+ 371us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:26:04.118576 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:26:04.118652 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 16:26:04.118766 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 16:26:04.119346 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:04.114472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43965 (local address 127.0.98.3:38465)
0929 16:26:04.114591 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:04.114595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:04.114700 (+ 105us) server_negotiation.cc:407] Connection header received
0929 16:26:04.114794 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:04.114797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:04.114853 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:04.114957 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:04.115454 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:04.116340 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:04.117526 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:04.117879 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:04.118009 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:04.118351 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:04.118456 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:04.118528 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:04.119148 (+ 620us) server_negotiation.cc:299] Negotiation successful
0929 16:26:04.119214 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 16:26:04.120117 2667 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 16:26:05.692830 2130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:05.692988 2130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:05.695513 2130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a: Sending DeleteTablet for 3 replicas of tablet dfac9a92ad764f6f8b113b1f500e058f
I20260929 16:26:05.696199 2268 tablet_service.cc:1514] Processing DeleteTablet for tablet dfac9a92ad764f6f8b113b1f500e058f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39060
I20260929 16:26:05.696277 2408 tablet_service.cc:1514] Processing DeleteTablet for tablet dfac9a92ad764f6f8b113b1f500e058f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58670
I20260929 16:26:05.696570 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet dfac9a92ad764f6f8b113b1f500e058f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35410
I20260929 16:26:05.696583 2727 tablet_replica.cc:330] T dfac9a92ad764f6f8b113b1f500e058f P 2a7e4632bf4d43df9a0d3f0011c522ac: stopping tablet replica
I20260929 16:26:05.696714 2727 raft_consensus.cc:2229] T dfac9a92ad764f6f8b113b1f500e058f P 2a7e4632bf4d43df9a0d3f0011c522ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:05.696838 2727 raft_consensus.cc:2258] T dfac9a92ad764f6f8b113b1f500e058f P 2a7e4632bf4d43df9a0d3f0011c522ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:05.697070 2729 tablet_replica.cc:330] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664: stopping tablet replica
I20260929 16:26:05.697068 2728 tablet_replica.cc:330] T dfac9a92ad764f6f8b113b1f500e058f P 469c940f7a7d40328acb8639ea30596d: stopping tablet replica
I20260929 16:26:05.697220 2728 raft_consensus.cc:2229] T dfac9a92ad764f6f8b113b1f500e058f P 469c940f7a7d40328acb8639ea30596d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:05.697371 2728 raft_consensus.cc:2258] T dfac9a92ad764f6f8b113b1f500e058f P 469c940f7a7d40328acb8639ea30596d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:05.697415 2729 raft_consensus.cc:2229] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:26:05.697650 2727 ts_tablet_manager.cc:1905] T dfac9a92ad764f6f8b113b1f500e058f P 2a7e4632bf4d43df9a0d3f0011c522ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:05.697824 2729 raft_consensus.cc:2258] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:05.698266 2728 ts_tablet_manager.cc:1905] T dfac9a92ad764f6f8b113b1f500e058f P 469c940f7a7d40328acb8639ea30596d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:05.699879 2729 ts_tablet_manager.cc:1905] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:05.700533 2727 ts_tablet_manager.cc:1918] T dfac9a92ad764f6f8b113b1f500e058f P 2a7e4632bf4d43df9a0d3f0011c522ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:26:05.700639 2727 log.cc:1196] T dfac9a92ad764f6f8b113b1f500e058f P 2a7e4632bf4d43df9a0d3f0011c522ac: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-0/wal/wals/dfac9a92ad764f6f8b113b1f500e058f
I20260929 16:26:05.700886 2727 ts_tablet_manager.cc:1939] T dfac9a92ad764f6f8b113b1f500e058f P 2a7e4632bf4d43df9a0d3f0011c522ac: Deleting consensus metadata
I20260929 16:26:05.701189 2728 ts_tablet_manager.cc:1918] T dfac9a92ad764f6f8b113b1f500e058f P 469c940f7a7d40328acb8639ea30596d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:26:05.701416 2728 log.cc:1196] T dfac9a92ad764f6f8b113b1f500e058f P 469c940f7a7d40328acb8639ea30596d: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/dfac9a92ad764f6f8b113b1f500e058f
I20260929 16:26:05.701777 2728 ts_tablet_manager.cc:1939] T dfac9a92ad764f6f8b113b1f500e058f P 469c940f7a7d40328acb8639ea30596d: Deleting consensus metadata
I20260929 16:26:05.701812 2115 catalog_manager.cc:4841] TS 2a7e4632bf4d43df9a0d3f0011c522ac (127.0.98.1:44297): tablet dfac9a92ad764f6f8b113b1f500e058f (table test-table [id=afa6dcd5725145778e4cd0ca0367454e]) successfully deleted
I20260929 16:26:05.702603 2117 catalog_manager.cc:4841] TS 469c940f7a7d40328acb8639ea30596d (127.0.98.2:38023): tablet dfac9a92ad764f6f8b113b1f500e058f (table test-table [id=afa6dcd5725145778e4cd0ca0367454e]) successfully deleted
I20260929 16:26:05.702880 2729 ts_tablet_manager.cc:1918] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:26:05.702975 2729 log.cc:1196] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/dfac9a92ad764f6f8b113b1f500e058f
I20260929 16:26:05.703231 2729 ts_tablet_manager.cc:1939] T dfac9a92ad764f6f8b113b1f500e058f P 4f645d147fa44b579755a51b53ff0664: Deleting consensus metadata
I20260929 16:26:05.703877 2116 catalog_manager.cc:4841] TS 4f645d147fa44b579755a51b53ff0664 (127.0.98.3:38465): tablet dfac9a92ad764f6f8b113b1f500e058f (table test-table [id=afa6dcd5725145778e4cd0ca0367454e]) successfully deleted
Sep 29 16:26:05 dist-test-slave-vq9v krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699165, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:26:05 dist-test-slave-vq9v krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699165, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:05.726435 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:05.716647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56764 (local address 127.0.98.62:34265)
0929 16:26:05.717070 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:05.717083 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:05.717195 (+ 112us) server_negotiation.cc:407] Connection header received
0929 16:26:05.717410 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:05.717413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:05.717464 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:05.717567 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:05.718268 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:05.718817 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:05.719658 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:05.719891 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:05.720019 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:05.722871 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:05.722875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:05.722878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:05.722916 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:05.724523 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:05.725020 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:05.725024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:05.725025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:05.725146 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:05.725477 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:05.725480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:05.725482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:05.725679 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:05.725825 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:05.726084 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 16:26:05.726242 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":92,"threads_started":1}
I20260929 16:26:05.729358 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:26:05.729830 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:26:05.734840 2268 tablet_service.cc:1467] Processing CreateTablet for tablet 051d0849bfa649a3b0cb30e91ed2155f (DEFAULT_TABLE table=test-table [id=cebbfb385a154c0abfda4de1a04ad212]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:05.734920 2408 tablet_service.cc:1467] Processing CreateTablet for tablet 051d0849bfa649a3b0cb30e91ed2155f (DEFAULT_TABLE table=test-table [id=cebbfb385a154c0abfda4de1a04ad212]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:05.735059 2541 tablet_service.cc:1467] Processing CreateTablet for tablet 051d0849bfa649a3b0cb30e91ed2155f (DEFAULT_TABLE table=test-table [id=cebbfb385a154c0abfda4de1a04ad212]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:05.735189 2408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 051d0849bfa649a3b0cb30e91ed2155f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:05.735189 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 051d0849bfa649a3b0cb30e91ed2155f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:05.735353 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 051d0849bfa649a3b0cb30e91ed2155f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:05.737558 2738 tablet_bootstrap.cc:492] T 051d0849bfa649a3b0cb30e91ed2155f P 4f645d147fa44b579755a51b53ff0664: Bootstrap starting.
I20260929 16:26:05.738222 2740 tablet_bootstrap.cc:492] T 051d0849bfa649a3b0cb30e91ed2155f P 469c940f7a7d40328acb8639ea30596d: Bootstrap starting.
I20260929 16:26:05.738447 2739 tablet_bootstrap.cc:492] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac: Bootstrap starting.
I20260929 16:26:05.738549 2738 tablet_bootstrap.cc:654] T 051d0849bfa649a3b0cb30e91ed2155f P 4f645d147fa44b579755a51b53ff0664: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:05.739174 2740 tablet_bootstrap.cc:654] T 051d0849bfa649a3b0cb30e91ed2155f P 469c940f7a7d40328acb8639ea30596d: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:05.739614 2739 tablet_bootstrap.cc:654] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:05.739889 2738 tablet_bootstrap.cc:492] T 051d0849bfa649a3b0cb30e91ed2155f P 4f645d147fa44b579755a51b53ff0664: No bootstrap required, opened a new log
I20260929 16:26:05.740015 2738 ts_tablet_manager.cc:1397] T 051d0849bfa649a3b0cb30e91ed2155f P 4f645d147fa44b579755a51b53ff0664: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:26:05.740315 2740 tablet_bootstrap.cc:492] T 051d0849bfa649a3b0cb30e91ed2155f P 469c940f7a7d40328acb8639ea30596d: No bootstrap required, opened a new log
I20260929 16:26:05.740422 2740 ts_tablet_manager.cc:1397] T 051d0849bfa649a3b0cb30e91ed2155f P 469c940f7a7d40328acb8639ea30596d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:26:05.740455 2738 raft_consensus.cc:348] T 051d0849bfa649a3b0cb30e91ed2155f P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:05.740566 2738 raft_consensus.cc:374] T 051d0849bfa649a3b0cb30e91ed2155f P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:05.740590 2738 raft_consensus.cc:729] T 051d0849bfa649a3b0cb30e91ed2155f P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f645d147fa44b579755a51b53ff0664, State: Initialized, Role: FOLLOWER
I20260929 16:26:05.740770 2738 consensus_queue.cc:260] T 051d0849bfa649a3b0cb30e91ed2155f P 4f645d147fa44b579755a51b53ff0664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:05.740834 2740 raft_consensus.cc:348] T 051d0849bfa649a3b0cb30e91ed2155f P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:05.740980 2740 raft_consensus.cc:374] T 051d0849bfa649a3b0cb30e91ed2155f P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:05.741042 2740 raft_consensus.cc:729] T 051d0849bfa649a3b0cb30e91ed2155f P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469c940f7a7d40328acb8639ea30596d, State: Initialized, Role: FOLLOWER
I20260929 16:26:05.741174 2738 ts_tablet_manager.cc:1428] T 051d0849bfa649a3b0cb30e91ed2155f P 4f645d147fa44b579755a51b53ff0664: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:26:05.741183 2740 consensus_queue.cc:260] T 051d0849bfa649a3b0cb30e91ed2155f P 469c940f7a7d40328acb8639ea30596d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:05.741988 2740 ts_tablet_manager.cc:1428] T 051d0849bfa649a3b0cb30e91ed2155f P 469c940f7a7d40328acb8639ea30596d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 16:26:05.743207 2739 tablet_bootstrap.cc:492] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac: No bootstrap required, opened a new log
I20260929 16:26:05.743366 2739 ts_tablet_manager.cc:1397] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 16:26:05.743929 2739 raft_consensus.cc:348] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:05.744131 2739 raft_consensus.cc:374] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:05.744218 2739 raft_consensus.cc:729] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7e4632bf4d43df9a0d3f0011c522ac, State: Initialized, Role: FOLLOWER
I20260929 16:26:05.744416 2739 consensus_queue.cc:260] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:05.744825 2739 ts_tablet_manager.cc:1428] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:26:05.974205 2744 raft_consensus.cc:482] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:26:05.974390 2744 raft_consensus.cc:504] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:05.974862 2744 leader_election.cc:288] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 469c940f7a7d40328acb8639ea30596d (127.0.98.2:38023), 4f645d147fa44b579755a51b53ff0664 (127.0.98.3:38465)
I20260929 16:26:05.975401 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "051d0849bfa649a3b0cb30e91ed2155f" candidate_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f645d147fa44b579755a51b53ff0664" is_pre_election: true
I20260929 16:26:05.975433 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "051d0849bfa649a3b0cb30e91ed2155f" candidate_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469c940f7a7d40328acb8639ea30596d" is_pre_election: true
I20260929 16:26:05.975576 2568 raft_consensus.cc:2454] T 051d0849bfa649a3b0cb30e91ed2155f P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a7e4632bf4d43df9a0d3f0011c522ac in term 0.
I20260929 16:26:05.975643 2428 raft_consensus.cc:2454] T 051d0849bfa649a3b0cb30e91ed2155f P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a7e4632bf4d43df9a0d3f0011c522ac in term 0.
I20260929 16:26:05.975950 2220 leader_election.cc:302] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a7e4632bf4d43df9a0d3f0011c522ac, 4f645d147fa44b579755a51b53ff0664; no voters:
I20260929 16:26:05.976130 2744 raft_consensus.cc:2781] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:26:05.976194 2744 raft_consensus.cc:482] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:26:05.976235 2744 raft_consensus.cc:3037] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:05.977247 2744 raft_consensus.cc:504] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:05.977631 2744 leader_election.cc:288] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac [CANDIDATE]: Term 1 election: Requested vote from peers 469c940f7a7d40328acb8639ea30596d (127.0.98.2:38023), 4f645d147fa44b579755a51b53ff0664 (127.0.98.3:38465)
I20260929 16:26:05.978008 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "051d0849bfa649a3b0cb30e91ed2155f" candidate_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469c940f7a7d40328acb8639ea30596d"
I20260929 16:26:05.978055 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "051d0849bfa649a3b0cb30e91ed2155f" candidate_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f645d147fa44b579755a51b53ff0664"
I20260929 16:26:05.978148 2428 raft_consensus.cc:3037] T 051d0849bfa649a3b0cb30e91ed2155f P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:05.978165 2568 raft_consensus.cc:3037] T 051d0849bfa649a3b0cb30e91ed2155f P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:05.979302 2568 raft_consensus.cc:2454] T 051d0849bfa649a3b0cb30e91ed2155f P 4f645d147fa44b579755a51b53ff0664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a7e4632bf4d43df9a0d3f0011c522ac in term 1.
I20260929 16:26:05.979323 2428 raft_consensus.cc:2454] T 051d0849bfa649a3b0cb30e91ed2155f P 469c940f7a7d40328acb8639ea30596d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a7e4632bf4d43df9a0d3f0011c522ac in term 1.
I20260929 16:26:05.979652 2221 leader_election.cc:302] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a7e4632bf4d43df9a0d3f0011c522ac, 469c940f7a7d40328acb8639ea30596d; no voters:
I20260929 16:26:05.979846 2744 raft_consensus.cc:2781] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:05.980139 2744 raft_consensus.cc:686] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac [term 1 LEADER]: Becoming Leader. State: Replica: 2a7e4632bf4d43df9a0d3f0011c522ac, State: Running, Role: LEADER
I20260929 16:26:05.980316 2744 consensus_queue.cc:237] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:05.982149 2130 catalog_manager.cc:5495] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a7e4632bf4d43df9a0d3f0011c522ac (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } health_report { overall_health: UNKNOWN } } }
I20260929 16:26:06.046363 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:06.042943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.0.98.1:44297)
0929 16:26:06.043248 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:06.043263 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:06.043274 (+ 11us) server_negotiation.cc:407] Connection header received
0929 16:26:06.043315 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:06.043317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:06.043358 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:06.043459 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:06.043740 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:06.044279 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:06.045157 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:06.045460 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:06.045587 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:06.045879 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:06.045963 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:06.046056 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:06.046138 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 16:26:06.046196 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":103,"threads_started":1}
I20260929 16:26:06.052773 2568 raft_consensus.cc:1261] T 051d0849bfa649a3b0cb30e91ed2155f P 4f645d147fa44b579755a51b53ff0664 [term 1 FOLLOWER]: Refusing update from remote peer 2a7e4632bf4d43df9a0d3f0011c522ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:06.052860 2428 raft_consensus.cc:1261] T 051d0849bfa649a3b0cb30e91ed2155f P 469c940f7a7d40328acb8639ea30596d [term 1 FOLLOWER]: Refusing update from remote peer 2a7e4632bf4d43df9a0d3f0011c522ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:06.053270 2745 consensus_queue.cc:1035] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:06.053449 2744 consensus_queue.cc:1035] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:06.087282 2129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:06.087468 2129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:06.089636 2129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a: Sending DeleteTablet for 3 replicas of tablet 051d0849bfa649a3b0cb30e91ed2155f
I20260929 16:26:06.090354 2268 tablet_service.cc:1514] Processing DeleteTablet for tablet 051d0849bfa649a3b0cb30e91ed2155f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39060
I20260929 16:26:06.090363 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet 051d0849bfa649a3b0cb30e91ed2155f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35410
I20260929 16:26:06.090363 2408 tablet_service.cc:1514] Processing DeleteTablet for tablet 051d0849bfa649a3b0cb30e91ed2155f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58670
I20260929 16:26:06.090601 2729 tablet_replica.cc:330] T 051d0849bfa649a3b0cb30e91ed2155f P 4f645d147fa44b579755a51b53ff0664: stopping tablet replica
I20260929 16:26:06.090623 2728 tablet_replica.cc:330] T 051d0849bfa649a3b0cb30e91ed2155f P 469c940f7a7d40328acb8639ea30596d: stopping tablet replica
I20260929 16:26:06.090638 2727 tablet_replica.cc:330] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac: stopping tablet replica
I20260929 16:26:06.090734 2729 raft_consensus.cc:2229] T 051d0849bfa649a3b0cb30e91ed2155f P 4f645d147fa44b579755a51b53ff0664 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:06.090734 2728 raft_consensus.cc:2229] T 051d0849bfa649a3b0cb30e91ed2155f P 469c940f7a7d40328acb8639ea30596d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:06.090804 2727 raft_consensus.cc:2229] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:26:06.090852 2729 raft_consensus.cc:2258] T 051d0849bfa649a3b0cb30e91ed2155f P 4f645d147fa44b579755a51b53ff0664 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:06.090852 2728 raft_consensus.cc:2258] T 051d0849bfa649a3b0cb30e91ed2155f P 469c940f7a7d40328acb8639ea30596d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:06.091025 2727 raft_consensus.cc:2258] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:06.091707 2729 ts_tablet_manager.cc:1905] T 051d0849bfa649a3b0cb30e91ed2155f P 4f645d147fa44b579755a51b53ff0664: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:06.091738 2727 ts_tablet_manager.cc:1905] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:06.092525 2129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:26:06.092999 2129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:26:06.094390 2729 ts_tablet_manager.cc:1918] T 051d0849bfa649a3b0cb30e91ed2155f P 4f645d147fa44b579755a51b53ff0664: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:26:06.094390 2727 ts_tablet_manager.cc:1918] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:26:06.094476 2727 log.cc:1196] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-0/wal/wals/051d0849bfa649a3b0cb30e91ed2155f
I20260929 16:26:06.094476 2729 log.cc:1196] T 051d0849bfa649a3b0cb30e91ed2155f P 4f645d147fa44b579755a51b53ff0664: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/051d0849bfa649a3b0cb30e91ed2155f
I20260929 16:26:06.094738 2727 ts_tablet_manager.cc:1939] T 051d0849bfa649a3b0cb30e91ed2155f P 2a7e4632bf4d43df9a0d3f0011c522ac: Deleting consensus metadata
I20260929 16:26:06.094738 2729 ts_tablet_manager.cc:1939] T 051d0849bfa649a3b0cb30e91ed2155f P 4f645d147fa44b579755a51b53ff0664: Deleting consensus metadata
I20260929 16:26:06.095466 2728 ts_tablet_manager.cc:1905] T 051d0849bfa649a3b0cb30e91ed2155f P 469c940f7a7d40328acb8639ea30596d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:06.097806 2728 ts_tablet_manager.cc:1918] T 051d0849bfa649a3b0cb30e91ed2155f P 469c940f7a7d40328acb8639ea30596d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:26:06.097898 2728 log.cc:1196] T 051d0849bfa649a3b0cb30e91ed2155f P 469c940f7a7d40328acb8639ea30596d: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/051d0849bfa649a3b0cb30e91ed2155f
I20260929 16:26:06.097994 2116 catalog_manager.cc:4841] TS 4f645d147fa44b579755a51b53ff0664 (127.0.98.3:38465): tablet 051d0849bfa649a3b0cb30e91ed2155f (table test-table [id=cebbfb385a154c0abfda4de1a04ad212]) successfully deleted
I20260929 16:26:06.098179 2728 ts_tablet_manager.cc:1939] T 051d0849bfa649a3b0cb30e91ed2155f P 469c940f7a7d40328acb8639ea30596d: Deleting consensus metadata
I20260929 16:26:06.098578 2115 catalog_manager.cc:4841] TS 2a7e4632bf4d43df9a0d3f0011c522ac (127.0.98.1:44297): tablet 051d0849bfa649a3b0cb30e91ed2155f (table test-table [id=cebbfb385a154c0abfda4de1a04ad212]) successfully deleted
I20260929 16:26:06.098634 2541 tablet_service.cc:1467] Processing CreateTablet for tablet bc9dee98e9c94cafadd804af26a9690d (DEFAULT_TABLE table=test-table [id=73d34e4700884db0b0288943cf4d7062]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:06.098924 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9dee98e9c94cafadd804af26a9690d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:06.099056 2268 tablet_service.cc:1467] Processing CreateTablet for tablet bc9dee98e9c94cafadd804af26a9690d (DEFAULT_TABLE table=test-table [id=73d34e4700884db0b0288943cf4d7062]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:06.099361 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9dee98e9c94cafadd804af26a9690d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:06.100739 2738 tablet_bootstrap.cc:492] T bc9dee98e9c94cafadd804af26a9690d P 4f645d147fa44b579755a51b53ff0664: Bootstrap starting.
I20260929 16:26:06.101595 2738 tablet_bootstrap.cc:654] T bc9dee98e9c94cafadd804af26a9690d P 4f645d147fa44b579755a51b53ff0664: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:06.101678 2117 catalog_manager.cc:4841] TS 469c940f7a7d40328acb8639ea30596d (127.0.98.2:38023): tablet 051d0849bfa649a3b0cb30e91ed2155f (table test-table [id=cebbfb385a154c0abfda4de1a04ad212]) successfully deleted
I20260929 16:26:06.101670 2408 tablet_service.cc:1467] Processing CreateTablet for tablet bc9dee98e9c94cafadd804af26a9690d (DEFAULT_TABLE table=test-table [id=73d34e4700884db0b0288943cf4d7062]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:06.101938 2408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9dee98e9c94cafadd804af26a9690d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:06.102643 2738 tablet_bootstrap.cc:492] T bc9dee98e9c94cafadd804af26a9690d P 4f645d147fa44b579755a51b53ff0664: No bootstrap required, opened a new log
I20260929 16:26:06.102751 2738 ts_tablet_manager.cc:1397] T bc9dee98e9c94cafadd804af26a9690d P 4f645d147fa44b579755a51b53ff0664: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 16:26:06.103093 2738 raft_consensus.cc:348] T bc9dee98e9c94cafadd804af26a9690d P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:06.103215 2738 raft_consensus.cc:374] T bc9dee98e9c94cafadd804af26a9690d P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:06.103260 2738 raft_consensus.cc:729] T bc9dee98e9c94cafadd804af26a9690d P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f645d147fa44b579755a51b53ff0664, State: Initialized, Role: FOLLOWER
I20260929 16:26:06.103394 2738 consensus_queue.cc:260] T bc9dee98e9c94cafadd804af26a9690d P 4f645d147fa44b579755a51b53ff0664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:06.103683 2738 ts_tablet_manager.cc:1428] T bc9dee98e9c94cafadd804af26a9690d P 4f645d147fa44b579755a51b53ff0664: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:26:06.104246 2740 tablet_bootstrap.cc:492] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d: Bootstrap starting.
I20260929 16:26:06.104704 2739 tablet_bootstrap.cc:492] T bc9dee98e9c94cafadd804af26a9690d P 2a7e4632bf4d43df9a0d3f0011c522ac: Bootstrap starting.
I20260929 16:26:06.105293 2740 tablet_bootstrap.cc:654] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:06.105552 2739 tablet_bootstrap.cc:654] T bc9dee98e9c94cafadd804af26a9690d P 2a7e4632bf4d43df9a0d3f0011c522ac: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:06.106652 2739 tablet_bootstrap.cc:492] T bc9dee98e9c94cafadd804af26a9690d P 2a7e4632bf4d43df9a0d3f0011c522ac: No bootstrap required, opened a new log
I20260929 16:26:06.106734 2740 tablet_bootstrap.cc:492] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d: No bootstrap required, opened a new log
I20260929 16:26:06.106774 2739 ts_tablet_manager.cc:1397] T bc9dee98e9c94cafadd804af26a9690d P 2a7e4632bf4d43df9a0d3f0011c522ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 16:26:06.106856 2740 ts_tablet_manager.cc:1397] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 16:26:06.107147 2739 raft_consensus.cc:348] T bc9dee98e9c94cafadd804af26a9690d P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:06.107239 2739 raft_consensus.cc:374] T bc9dee98e9c94cafadd804af26a9690d P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:06.107259 2739 raft_consensus.cc:729] T bc9dee98e9c94cafadd804af26a9690d P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7e4632bf4d43df9a0d3f0011c522ac, State: Initialized, Role: FOLLOWER
I20260929 16:26:06.107319 2740 raft_consensus.cc:348] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:06.107419 2740 raft_consensus.cc:374] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:06.107394 2739 consensus_queue.cc:260] T bc9dee98e9c94cafadd804af26a9690d P 2a7e4632bf4d43df9a0d3f0011c522ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:06.107445 2740 raft_consensus.cc:729] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469c940f7a7d40328acb8639ea30596d, State: Initialized, Role: FOLLOWER
I20260929 16:26:06.107582 2740 consensus_queue.cc:260] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:06.107668 2739 ts_tablet_manager.cc:1428] T bc9dee98e9c94cafadd804af26a9690d P 2a7e4632bf4d43df9a0d3f0011c522ac: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:26:06.107896 2740 ts_tablet_manager.cc:1428] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:26:06.207211 2708 raft_consensus.cc:482] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:26:06.207458 2708 raft_consensus.cc:504] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:06.207958 2708 leader_election.cc:288] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a7e4632bf4d43df9a0d3f0011c522ac (127.0.98.1:44297), 4f645d147fa44b579755a51b53ff0664 (127.0.98.3:38465)
I20260929 16:26:06.208508 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9dee98e9c94cafadd804af26a9690d" candidate_uuid: "469c940f7a7d40328acb8639ea30596d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" is_pre_election: true
I20260929 16:26:06.208621 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9dee98e9c94cafadd804af26a9690d" candidate_uuid: "469c940f7a7d40328acb8639ea30596d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f645d147fa44b579755a51b53ff0664" is_pre_election: true
I20260929 16:26:06.208676 2287 raft_consensus.cc:2454] T bc9dee98e9c94cafadd804af26a9690d P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 469c940f7a7d40328acb8639ea30596d in term 0.
I20260929 16:26:06.208793 2568 raft_consensus.cc:2454] T bc9dee98e9c94cafadd804af26a9690d P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 469c940f7a7d40328acb8639ea30596d in term 0.
I20260929 16:26:06.209064 2359 leader_election.cc:302] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a7e4632bf4d43df9a0d3f0011c522ac, 469c940f7a7d40328acb8639ea30596d; no voters:
I20260929 16:26:06.210067 2708 raft_consensus.cc:2781] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:26:06.210157 2708 raft_consensus.cc:482] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:26:06.210218 2708 raft_consensus.cc:3037] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:06.211236 2708 raft_consensus.cc:504] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:06.211633 2708 leader_election.cc:288] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d [CANDIDATE]: Term 1 election: Requested vote from peers 2a7e4632bf4d43df9a0d3f0011c522ac (127.0.98.1:44297), 4f645d147fa44b579755a51b53ff0664 (127.0.98.3:38465)
I20260929 16:26:06.212013 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9dee98e9c94cafadd804af26a9690d" candidate_uuid: "469c940f7a7d40328acb8639ea30596d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac"
I20260929 16:26:06.212145 2287 raft_consensus.cc:3037] T bc9dee98e9c94cafadd804af26a9690d P 2a7e4632bf4d43df9a0d3f0011c522ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:06.212081 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9dee98e9c94cafadd804af26a9690d" candidate_uuid: "469c940f7a7d40328acb8639ea30596d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f645d147fa44b579755a51b53ff0664"
I20260929 16:26:06.212262 2568 raft_consensus.cc:3037] T bc9dee98e9c94cafadd804af26a9690d P 4f645d147fa44b579755a51b53ff0664 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:06.213140 2287 raft_consensus.cc:2454] T bc9dee98e9c94cafadd804af26a9690d P 2a7e4632bf4d43df9a0d3f0011c522ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 469c940f7a7d40328acb8639ea30596d in term 1.
I20260929 16:26:06.213171 2568 raft_consensus.cc:2454] T bc9dee98e9c94cafadd804af26a9690d P 4f645d147fa44b579755a51b53ff0664 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 469c940f7a7d40328acb8639ea30596d in term 1.
I20260929 16:26:06.213536 2360 leader_election.cc:302] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 469c940f7a7d40328acb8639ea30596d, 4f645d147fa44b579755a51b53ff0664; no voters:
I20260929 16:26:06.213708 2708 raft_consensus.cc:2781] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:06.213815 2708 raft_consensus.cc:686] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d [term 1 LEADER]: Becoming Leader. State: Replica: 469c940f7a7d40328acb8639ea30596d, State: Running, Role: LEADER
I20260929 16:26:06.213972 2708 consensus_queue.cc:237] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } }
I20260929 16:26:06.215657 2131 catalog_manager.cc:5495] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d reported cstate change: term changed from 0 to 1, leader changed from <none> to 469c940f7a7d40328acb8639ea30596d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "469c940f7a7d40328acb8639ea30596d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "469c940f7a7d40328acb8639ea30596d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 } health_report { overall_health: UNKNOWN } } }
I20260929 16:26:06.239216 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:06.235766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60374 (local address 127.0.98.2:38023)
0929 16:26:06.236095 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:06.236102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:06.236116 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:26:06.236156 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:06.236158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:06.236202 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:06.236278 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:06.236478 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:06.237062 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:06.237997 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:06.238254 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:06.238345 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:06.238652 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:06.238727 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:06.238785 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:06.238936 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 16:26:06.239025 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":136,"threads_started":1}
I20260929 16:26:06.248366 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:06.245485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51705 (local address 127.0.98.2:38023)
0929 16:26:06.245605 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:06.245608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:06.245657 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:26:06.245806 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:06.245808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:06.245847 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:06.245915 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:06.246137 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:06.246644 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:06.247458 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:06.247692 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:06.247779 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:06.247989 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:06.248062 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:06.248120 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:06.248201 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 16:26:06.248243 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:26:06.248378 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:06.245315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38023 (local address 127.0.98.2:51705)
0929 16:26:06.245592 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:06.245606 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:06.245696 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:06.245940 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:06.245941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:06.245948 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:26:06.246033 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:06.246037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:06.246733 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:06.246735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:06.247338 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:06.247346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:06.247802 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:06.247804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:06.247867 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:06.247891 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:26:06.248140 (+ 249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:26:06.248144 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 16:26:06.248194 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 16:26:06.248260 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":112,"threads_started":1}
I20260929 16:26:06.254513 2568 raft_consensus.cc:1261] T bc9dee98e9c94cafadd804af26a9690d P 4f645d147fa44b579755a51b53ff0664 [term 1 FOLLOWER]: Refusing update from remote peer 469c940f7a7d40328acb8639ea30596d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:06.254551 2287 raft_consensus.cc:1261] T bc9dee98e9c94cafadd804af26a9690d P 2a7e4632bf4d43df9a0d3f0011c522ac [term 1 FOLLOWER]: Refusing update from remote peer 469c940f7a7d40328acb8639ea30596d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:06.254958 2708 consensus_queue.cc:1035] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f645d147fa44b579755a51b53ff0664" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:06.255119 2714 consensus_queue.cc:1035] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a7e4632bf4d43df9a0d3f0011c522ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:06.258271 2706 mvcc.cc:205] Tried to move back new op lower bound from 7334703784975523840 to 7334703784813867008. Current Snapshot: MvccSnapshot[applied={T|T < 7334703784975523840}]
W20260929 16:26:06.269502 2706 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 16:26:07.828177 2130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:07.828369 2130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:07.830993 2130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9593a903c18044deb7e38a0c478e601a: Sending DeleteTablet for 3 replicas of tablet bc9dee98e9c94cafadd804af26a9690d
I20260929 16:26:07.831662 2268 tablet_service.cc:1514] Processing DeleteTablet for tablet bc9dee98e9c94cafadd804af26a9690d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39060
I20260929 16:26:07.831806 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet bc9dee98e9c94cafadd804af26a9690d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35410
I20260929 16:26:07.832008 2407 tablet_service.cc:1514] Processing DeleteTablet for tablet bc9dee98e9c94cafadd804af26a9690d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58670
I20260929 16:26:07.832144 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 2203
I20260929 16:26:07.832152 2784 tablet_replica.cc:330] T bc9dee98e9c94cafadd804af26a9690d P 2a7e4632bf4d43df9a0d3f0011c522ac: stopping tablet replica
I20260929 16:26:07.833107 2786 tablet_replica.cc:330] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d: stopping tablet replica
I20260929 16:26:07.833334 2786 raft_consensus.cc:2229] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:26:07.833837 2786 raft_consensus.cc:2258] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:07.834470 2786 ts_tablet_manager.cc:1905] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:07.836619 2786 ts_tablet_manager.cc:1918] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:26:07.836687 2786 log.cc:1196] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/bc9dee98e9c94cafadd804af26a9690d
I20260929 16:26:07.836966 2786 ts_tablet_manager.cc:1939] T bc9dee98e9c94cafadd804af26a9690d P 469c940f7a7d40328acb8639ea30596d: Deleting consensus metadata
I20260929 16:26:07.836961 2785 tablet_replica.cc:330] T bc9dee98e9c94cafadd804af26a9690d P 4f645d147fa44b579755a51b53ff0664: stopping tablet replica
I20260929 16:26:07.837133 2785 raft_consensus.cc:2229] T bc9dee98e9c94cafadd804af26a9690d P 4f645d147fa44b579755a51b53ff0664 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:07.837314 2785 raft_consensus.cc:2258] T bc9dee98e9c94cafadd804af26a9690d P 4f645d147fa44b579755a51b53ff0664 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:07.837747 2117 catalog_manager.cc:4841] TS 469c940f7a7d40328acb8639ea30596d (127.0.98.2:38023): tablet bc9dee98e9c94cafadd804af26a9690d (table test-table [id=73d34e4700884db0b0288943cf4d7062]) successfully deleted
I20260929 16:26:07.838070 2785 ts_tablet_manager.cc:1905] T bc9dee98e9c94cafadd804af26a9690d P 4f645d147fa44b579755a51b53ff0664: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 16:26:07.841312 2115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44297 (error 108)
I20260929 16:26:07.841498 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 2343
I20260929 16:26:07.842192 2785 ts_tablet_manager.cc:1918] T bc9dee98e9c94cafadd804af26a9690d P 4f645d147fa44b579755a51b53ff0664: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:26:07.842311 2785 log.cc:1196] T bc9dee98e9c94cafadd804af26a9690d P 4f645d147fa44b579755a51b53ff0664: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/bc9dee98e9c94cafadd804af26a9690d
I20260929 16:26:07.842651 2785 ts_tablet_manager.cc:1939] T bc9dee98e9c94cafadd804af26a9690d P 4f645d147fa44b579755a51b53ff0664: Deleting consensus metadata
I20260929 16:26:07.843523 2116 catalog_manager.cc:4841] TS 4f645d147fa44b579755a51b53ff0664 (127.0.98.3:38465): tablet bc9dee98e9c94cafadd804af26a9690d (table test-table [id=73d34e4700884db0b0288943cf4d7062]) successfully deleted
I20260929 16:26:07.845573 2788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:07.842743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44297 (local address 127.0.0.1:34236)
0929 16:26:07.845416 (+ 2673us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:07.845485 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2598,"thread_start_us":83,"threads_started":1}
W20260929 16:26:07.845742 2115 catalog_manager.cc:4584] TS 2a7e4632bf4d43df9a0d3f0011c522ac (127.0.98.1:44297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bc9dee98e9c94cafadd804af26a9690d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44297: connect: Connection refused (error 111)
I20260929 16:26:07.849862 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 2483
I20260929 16:26:07.857651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 2099
2026-09-29T16:26:07Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7008 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:26:07 dist-test-slave-vq9v krb5kdc[2791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:26:07 dist-test-slave-vq9v krb5kdc[2791](info): set up 2 sockets
Sep 29 16:26:07 dist-test-slave-vq9v krb5kdc[2791](info): commencing operation
krb5kdc: starting...
W20260929 16:26:08.965445 392 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:26:09 dist-test-slave-vq9v krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:26:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:26:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:26:09.092906 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37123
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:09.205564 2804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:09.205909 2804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:09.205988 2804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:09.210125 2804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:26:09.210242 2804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:09.210279 2804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:09.210319 2804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:26:09.210351 2804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:26:09.216475 2804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:09.217677 2804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:09.218730 2804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:09.227965 2810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:09.227998 2804 server_base.cc:1034] running on GCE node
W20260929 16:26:09.228070 2809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:09.228178 2812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:09.228904 2804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:09.230185 2804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:09.231364 2804 hybrid_clock.cc:648] HybridClock initialized: now 1790699169231346 us; error 61 us; skew 500 ppm
Sep 29 16:26:09 dist-test-slave-vq9v krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:09.234433 2804 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:26:09.235363 2804 webserver.cc:460] Webserver started at http://127.0.98.62:44155/ using document root <none> and password file <none>
I20260929 16:26:09.235929 2804 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:09.235991 2804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:09.236271 2804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:09.238474 2804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "a2536e3c57ff46fab8d6d56a9bf1c8c6"
format_stamp: "Formatted at 2026-09-29 16:26:09 on dist-test-slave-vq9v"
server_key: "d63fc2d31d31092eaa7dd25241097eb2"
server_key_iv: "e165de03f29106462189b64730c3f368"
server_key_version: "encryptionkey@0"
I20260929 16:26:09.239061 2804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "a2536e3c57ff46fab8d6d56a9bf1c8c6"
format_stamp: "Formatted at 2026-09-29 16:26:09 on dist-test-slave-vq9v"
server_key: "d63fc2d31d31092eaa7dd25241097eb2"
server_key_iv: "e165de03f29106462189b64730c3f368"
server_key_version: "encryptionkey@0"
I20260929 16:26:09.242897 2804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:26:09.245476 2819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:09.246778 2804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:26:09.246887 2804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "a2536e3c57ff46fab8d6d56a9bf1c8c6"
format_stamp: "Formatted at 2026-09-29 16:26:09 on dist-test-slave-vq9v"
server_key: "d63fc2d31d31092eaa7dd25241097eb2"
server_key_iv: "e165de03f29106462189b64730c3f368"
server_key_version: "encryptionkey@0"
I20260929 16:26:09.247027 2804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:09.276935 2804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:09.280347 2804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:09.280573 2804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:09.287900 2804 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37123
I20260929 16:26:09.287906 2871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37123 every 8 connection(s)
I20260929 16:26:09.288928 2804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:26:09.289850 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 2804
I20260929 16:26:09.290052 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:26:09.290334 392 external_mini_cluster.cc:1428] Setting key fc15e8f9371b23048057f8786b235498
I20260929 16:26:09.292237 2872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:09.298173 2872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6: Bootstrap starting.
Sep 29 16:26:09 dist-test-slave-vq9v krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:09.300592 2872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:09.301507 2872 log.cc:824] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:09.303522 2872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6: No bootstrap required, opened a new log
I20260929 16:26:09.305459 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:09.291726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.0.98.62:37123)
0929 16:26:09.292217 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:09.292231 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:09.292270 (+ 39us) server_negotiation.cc:407] Connection header received
0929 16:26:09.293105 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:09.293123 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:09.293558 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:09.293995 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:09.294761 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:09.295759 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:09.296684 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:09.297189 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:09.297419 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:09.300222 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:09.300227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:09.300241 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:09.300271 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:09.302204 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:09.302717 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:09.302722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:09.302727 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:09.302840 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:09.303155 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:09.303159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:09.303160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:09.303612 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:09.303817 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:09.304124 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 16:26:09.304438 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":158,"threads_started":1}
I20260929 16:26:09.306941 2872 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2536e3c57ff46fab8d6d56a9bf1c8c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37123 } }
I20260929 16:26:09.307229 2872 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:09.307286 2872 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2536e3c57ff46fab8d6d56a9bf1c8c6, State: Initialized, Role: FOLLOWER
I20260929 16:26:09.307771 2872 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2536e3c57ff46fab8d6d56a9bf1c8c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37123 } }
I20260929 16:26:09.307927 2872 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:26:09.308007 2872 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:26:09.308090 2872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:09.309049 2872 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2536e3c57ff46fab8d6d56a9bf1c8c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37123 } }
I20260929 16:26:09.309422 2872 leader_election.cc:302] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2536e3c57ff46fab8d6d56a9bf1c8c6; no voters:
I20260929 16:26:09.309707 2872 leader_election.cc:288] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:26:09.309860 2877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:09.310209 2877 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6 [term 1 LEADER]: Becoming Leader. State: Replica: a2536e3c57ff46fab8d6d56a9bf1c8c6, State: Running, Role: LEADER
I20260929 16:26:09.310583 2877 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2536e3c57ff46fab8d6d56a9bf1c8c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37123 } }
I20260929 16:26:09.311342 2872 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:26:09.312475 2879 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2536e3c57ff46fab8d6d56a9bf1c8c6. Latest consensus state: current_term: 1 leader_uuid: "a2536e3c57ff46fab8d6d56a9bf1c8c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2536e3c57ff46fab8d6d56a9bf1c8c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37123 } } }
I20260929 16:26:09.312654 2879 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:09.312511 2878 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2536e3c57ff46fab8d6d56a9bf1c8c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2536e3c57ff46fab8d6d56a9bf1c8c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37123 } } }
I20260929 16:26:09.312759 2878 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:09.313107 2886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:26:09.316154 2886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:26:09.321973 2886 catalog_manager.cc:1347] Generated new cluster ID: 8b065f970b80456a830f69954b5b754b
I20260929 16:26:09.322100 2886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:26:09.330246 2886 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:26:09.331542 2886 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:26:09.341650 2886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2536e3c57ff46fab8d6d56a9bf1c8c6: Generated new TSK 0
I20260929 16:26:09.342410 2886 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 16:26:09.378513 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37123
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:09.495859 2898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:09.496160 2898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:09.496207 2898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:09.500401 2898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:09.500483 2898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:09.500545 2898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:26:09.506165 2898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:09.507231 2898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:09.508244 2898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:09.518890 2903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:09.518975 2898 server_base.cc:1034] running on GCE node
W20260929 16:26:09.518883 2906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:09.518890 2904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:09.519655 2898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:09.520469 2898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:09.521682 2898 hybrid_clock.cc:648] HybridClock initialized: now 1790699169521660 us; error 37 us; skew 500 ppm
Sep 29 16:26:09 dist-test-slave-vq9v krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:09.524842 2898 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:26:09.525908 2898 webserver.cc:460] Webserver started at http://127.0.98.1:35551/ using document root <none> and password file <none>
I20260929 16:26:09.526485 2898 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:09.526549 2898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:09.526736 2898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:09.528929 2898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "53984e358c6947c2882c34c8efccbfd0"
format_stamp: "Formatted at 2026-09-29 16:26:09 on dist-test-slave-vq9v"
server_key: "90846f22486c8308df616e4206da70b9"
server_key_iv: "028c47a35b51008efad3a2b6449758a1"
server_key_version: "encryptionkey@0"
I20260929 16:26:09.529601 2898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "53984e358c6947c2882c34c8efccbfd0"
format_stamp: "Formatted at 2026-09-29 16:26:09 on dist-test-slave-vq9v"
server_key: "90846f22486c8308df616e4206da70b9"
server_key_iv: "028c47a35b51008efad3a2b6449758a1"
server_key_version: "encryptionkey@0"
I20260929 16:26:09.534682 2898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:26:09.537660 2913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:09.539023 2898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:26:09.539182 2898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "53984e358c6947c2882c34c8efccbfd0"
format_stamp: "Formatted at 2026-09-29 16:26:09 on dist-test-slave-vq9v"
server_key: "90846f22486c8308df616e4206da70b9"
server_key_iv: "028c47a35b51008efad3a2b6449758a1"
server_key_version: "encryptionkey@0"
I20260929 16:26:09.539310 2898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:09.554838 2898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:09.558449 2898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:09.558688 2898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:09.559335 2898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:09.560217 2898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:09.560290 2898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:09.560365 2898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:09.560410 2898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:09.569485 2898 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42461
I20260929 16:26:09.569504 3026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42461 every 8 connection(s)
I20260929 16:26:09.570423 2898 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:26:09.575673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 2898
I20260929 16:26:09.575821 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:26:09.576151 392 external_mini_cluster.cc:1428] Setting key baae45086246a922f54b44682cf05a93
Sep 29 16:26:09 dist-test-slave-vq9v krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 16:26:09.585567 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:09.572470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53891 (local address 127.0.98.62:37123)
0929 16:26:09.572633 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:09.572637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:09.573608 (+ 971us) server_negotiation.cc:407] Connection header received
0929 16:26:09.575025 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:09.575029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:09.575087 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:09.575168 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:09.576552 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:09.577328 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:09.578541 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:09.578803 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:09.578924 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:09.581619 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:09.581622 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:09.581624 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:09.581649 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:09.583279 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:09.583849 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:09.583853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:09.583855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:09.583960 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:09.584359 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:09.584363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:09.584365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:09.584573 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:09.584679 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:09.585240 (+ 561us) server_negotiation.cc:299] Negotiation successful
0929 16:26:09.585384 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:26:09.586547 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:09.572903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37123 (local address 127.0.98.1:53891)
0929 16:26:09.573426 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:09.573472 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:09.574800 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:09.575350 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:09.575356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:09.575740 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:09.576387 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:09.576404 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:09.577468 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:09.577476 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:09.578403 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:09.578412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:09.578973 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:09.578977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:09.579156 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:09.579815 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:09.579832 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:09.581401 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:09.583411 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:09.583417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:09.583435 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:09.583731 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:09.584106 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:09.584109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:09.584110 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:09.584254 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:09.584690 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:09.584699 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:09.584971 (+ 272us) client_negotiation.cc:769] Sending connection context
0929 16:26:09.585181 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 16:26:09.585575 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 16:26:09.587956 3027 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37123
I20260929 16:26:09.588268 3027 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:09.588802 3027 heartbeater.cc:511] Master 127.0.98.62:37123 requested a full tablet report, sending...
I20260929 16:26:09.590672 2836 ts_manager.cc:194] Registered new tserver with Master: 53984e358c6947c2882c34c8efccbfd0 (127.0.98.1:42461)
I20260929 16:26:09.591871 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 16:26:09.608573 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37123
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:09.724205 3032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:09.724562 3032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:09.724614 3032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:09.729568 3032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:09.729671 3032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:09.729745 3032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:26:09.736527 3032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:09.737738 3032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:09.738830 3032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:09.749818 3037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:09.749887 3040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:09.749943 3038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:09.750067 3032 server_base.cc:1034] running on GCE node
I20260929 16:26:09.750519 3032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:09.751291 3032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:09.752511 3032 hybrid_clock.cc:648] HybridClock initialized: now 1790699169752476 us; error 48 us; skew 500 ppm
Sep 29 16:26:09 dist-test-slave-vq9v krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:09.755878 3032 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:26:09.757057 3032 webserver.cc:460] Webserver started at http://127.0.98.2:34101/ using document root <none> and password file <none>
I20260929 16:26:09.757753 3032 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:09.757851 3032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:09.758081 3032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:09.760365 3032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "98d8de7a4a844aa18f75b796ecd7b762"
format_stamp: "Formatted at 2026-09-29 16:26:09 on dist-test-slave-vq9v"
server_key: "002df644c4d42bddbccc01d4cf05f074"
server_key_iv: "271b77d8f8a72eca89f210be414f3f82"
server_key_version: "encryptionkey@0"
I20260929 16:26:09.760998 3032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "98d8de7a4a844aa18f75b796ecd7b762"
format_stamp: "Formatted at 2026-09-29 16:26:09 on dist-test-slave-vq9v"
server_key: "002df644c4d42bddbccc01d4cf05f074"
server_key_iv: "271b77d8f8a72eca89f210be414f3f82"
server_key_version: "encryptionkey@0"
I20260929 16:26:09.765149 3032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 16:26:09.767917 3047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:09.769362 3032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:09.769518 3032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "98d8de7a4a844aa18f75b796ecd7b762"
format_stamp: "Formatted at 2026-09-29 16:26:09 on dist-test-slave-vq9v"
server_key: "002df644c4d42bddbccc01d4cf05f074"
server_key_iv: "271b77d8f8a72eca89f210be414f3f82"
server_key_version: "encryptionkey@0"
I20260929 16:26:09.769649 3032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:09.787829 3032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:09.791381 3032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:09.791604 3032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:09.792232 3032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:09.793107 3032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:09.793177 3032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:09.793270 3032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:09.793329 3032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:09.803431 3032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46335
I20260929 16:26:09.803447 3160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46335 every 8 connection(s)
I20260929 16:26:09.804450 3032 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:26:09.806205 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 3032
I20260929 16:26:09.806315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:26:09.806636 392 external_mini_cluster.cc:1428] Setting key 2a07dc6eeefe01f796e62bfee52fda5e
Sep 29 16:26:09 dist-test-slave-vq9v krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 16:26:09.820938 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:09.806405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38155 (local address 127.0.98.62:37123)
0929 16:26:09.806584 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:09.806590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:09.807555 (+ 965us) server_negotiation.cc:407] Connection header received
0929 16:26:09.808991 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:09.808994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:09.809050 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:09.809159 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:09.810930 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:09.811485 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:09.813648 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:09.813921 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:09.814013 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:09.817111 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:09.817114 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:09.817117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:09.817143 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:09.818489 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:09.819099 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:09.819103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:09.819104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:09.819189 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:09.819673 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:09.819679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:09.819684 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:09.819943 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:09.820064 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:09.820632 (+ 568us) server_negotiation.cc:299] Negotiation successful
0929 16:26:09.820768 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:26:09.821955 3163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:09.806896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37123 (local address 127.0.98.2:38155)
0929 16:26:09.807386 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:09.807441 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:09.808754 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:09.809376 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:09.809384 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:09.809988 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:09.810728 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:09.810751 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:09.811648 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:09.811655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:09.813502 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:09.813518 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:09.814115 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:09.814121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:09.814357 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:09.815051 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:09.815077 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:09.816911 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:09.818660 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:09.818666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:09.818685 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:09.818972 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:09.819324 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:09.819326 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:09.819327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:09.819516 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:09.820072 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:09.820082 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:09.820361 (+ 279us) client_negotiation.cc:769] Sending connection context
0929 16:26:09.820567 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 16:26:09.820929 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":146,"threads_started":1}
I20260929 16:26:09.823231 3161 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37123
I20260929 16:26:09.823537 3161 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:09.824249 3161 heartbeater.cc:511] Master 127.0.98.62:37123 requested a full tablet report, sending...
I20260929 16:26:09.825647 2836 ts_manager.cc:194] Registered new tserver with Master: 98d8de7a4a844aa18f75b796ecd7b762 (127.0.98.2:46335)
I20260929 16:26:09.826256 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 16:26:09.841673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37123
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:09.960644 3166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:09.960979 3166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:09.961033 3166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:09.965652 3166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:09.965732 3166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:09.965794 3166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:26:09.971558 3166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:09.972826 3166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:09.974261 3166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:09.984853 3172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:09.984829 3174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:09.984862 3171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:09.985074 3166 server_base.cc:1034] running on GCE node
I20260929 16:26:09.985607 3166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:09.986315 3166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:09.987500 3166 hybrid_clock.cc:648] HybridClock initialized: now 1790699169987476 us; error 39 us; skew 500 ppm
Sep 29 16:26:09 dist-test-slave-vq9v krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:09.990651 3166 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:26:09.991648 3166 webserver.cc:460] Webserver started at http://127.0.98.3:45861/ using document root <none> and password file <none>
I20260929 16:26:09.992254 3166 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:09.992347 3166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:09.992588 3166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:09.994845 3166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "031ee7f7dc31419ab6a718df2baca3d8"
format_stamp: "Formatted at 2026-09-29 16:26:09 on dist-test-slave-vq9v"
server_key: "eb5c471dbc689544ccf9f2dfc9a8c337"
server_key_iv: "d12b2587e8c608b3e074df8ff6191cdc"
server_key_version: "encryptionkey@0"
I20260929 16:26:09.995469 3166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "031ee7f7dc31419ab6a718df2baca3d8"
format_stamp: "Formatted at 2026-09-29 16:26:09 on dist-test-slave-vq9v"
server_key: "eb5c471dbc689544ccf9f2dfc9a8c337"
server_key_iv: "d12b2587e8c608b3e074df8ff6191cdc"
server_key_version: "encryptionkey@0"
I20260929 16:26:09.999712 3166 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:26:10.002898 3181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:10.004490 3166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:26:10.004702 3166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "031ee7f7dc31419ab6a718df2baca3d8"
format_stamp: "Formatted at 2026-09-29 16:26:09 on dist-test-slave-vq9v"
server_key: "eb5c471dbc689544ccf9f2dfc9a8c337"
server_key_iv: "d12b2587e8c608b3e074df8ff6191cdc"
server_key_version: "encryptionkey@0"
I20260929 16:26:10.004858 3166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:10.019549 3166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:10.023200 3166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:10.023445 3166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:10.024076 3166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:10.024989 3166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:10.025064 3166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:10.025137 3166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:10.025190 3166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:10.034977 3166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44701
I20260929 16:26:10.035012 3294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44701 every 8 connection(s)
I20260929 16:26:10.035966 3166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:26:10.039162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 3166
I20260929 16:26:10.039289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:26:10.039599 392 external_mini_cluster.cc:1428] Setting key c1766d379642bf6ee6d3d8f5e382e91d
Sep 29 16:26:10 dist-test-slave-vq9v krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:10.052597 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:10.038089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51403 (local address 127.0.98.62:37123)
0929 16:26:10.038245 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:10.038249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:10.039317 (+ 1068us) server_negotiation.cc:407] Connection header received
0929 16:26:10.041515 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:10.041518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:10.041574 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:10.041641 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:10.043179 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:10.043768 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:10.045069 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:10.045355 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:10.045442 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:10.048260 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:10.048263 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:10.048266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:10.048293 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:10.049888 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:10.050626 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:10.050629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:10.050631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:10.050715 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:10.051149 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:10.051156 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:10.051159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:10.051409 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:10.051489 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:10.052282 (+ 793us) server_negotiation.cc:299] Negotiation successful
0929 16:26:10.052423 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:26:10.053736 3297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:10.038551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37123 (local address 127.0.98.3:51403)
0929 16:26:10.039117 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:10.039166 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:10.041247 (+ 2081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:10.041863 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:10.041875 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:10.042372 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:10.043032 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:10.043048 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:10.043907 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:10.043910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:10.044940 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:10.044950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:10.045517 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:10.045520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:10.045673 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:10.046407 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:10.046425 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:10.048037 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:10.050055 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:10.050061 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:10.050078 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:10.050517 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:10.050839 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:10.050841 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:10.050843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:10.051016 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:10.051538 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:10.051544 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:10.051990 (+ 446us) client_negotiation.cc:769] Sending connection context
0929 16:26:10.052191 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 16:26:10.052532 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":124,"threads_started":1}
I20260929 16:26:10.054949 3295 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37123
I20260929 16:26:10.055190 3295 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:10.055739 3295 heartbeater.cc:511] Master 127.0.98.62:37123 requested a full tablet report, sending...
I20260929 16:26:10.056784 2836 ts_manager.cc:194] Registered new tserver with Master: 031ee7f7dc31419ab6a718df2baca3d8 (127.0.98.3:44701)
I20260929 16:26:10.057449 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51403
I20260929 16:26:10.066694 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 16:26:10.080417 2875 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:10.078800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43664 (local address 127.0.98.62:37123)
0929 16:26:10.079011 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:10.079015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:10.079099 (+ 84us) server_negotiation.cc:407] Connection header received
0929 16:26:10.079225 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:10.079228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:10.079275 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:10.079350 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:10.080253 (+ 903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43664: BlockingRecv error: recv got EOF from 127.0.0.1:43664 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 16:26:10.081820 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 2898
I20260929 16:26:10.087574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 3032
I20260929 16:26:10.093140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 3166
I20260929 16:26:10.099141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 2804
2026-09-29T16:26:10Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2239 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:26:10 dist-test-slave-vq9v krb5kdc[3307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:26:10 dist-test-slave-vq9v krb5kdc[3307](info): set up 2 sockets
Sep 29 16:26:10 dist-test-slave-vq9v krb5kdc[3307](info): commencing operation
krb5kdc: starting...
W20260929 16:26:11.165310 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:26:11 dist-test-slave-vq9v krb5kdc[3307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:26:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:26:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:26:11.293049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33565
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:11.405351 3320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:11.405731 3320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:11.405818 3320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:11.410138 3320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:26:11.410266 3320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:11.410303 3320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:11.410344 3320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:26:11.410377 3320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:26:11.416384 3320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:11.417713 3320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:11.418874 3320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:11.428170 3328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:11.428182 3325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:11.428232 3320 server_base.cc:1034] running on GCE node
W20260929 16:26:11.428167 3326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:11.429113 3320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:11.430343 3320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:11.431545 3320 hybrid_clock.cc:648] HybridClock initialized: now 1790699171431518 us; error 35 us; skew 500 ppm
Sep 29 16:26:11 dist-test-slave-vq9v krb5kdc[3307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:11.434909 3320 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:26:11.435976 3320 webserver.cc:460] Webserver started at http://127.0.98.62:33051/ using document root <none> and password file <none>
I20260929 16:26:11.436579 3320 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:11.436646 3320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:11.436895 3320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:11.439179 3320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "9df102e2799d4b8d914e61b610da2f56"
format_stamp: "Formatted at 2026-09-29 16:26:11 on dist-test-slave-vq9v"
server_key: "f51950ec0744510410fcc05ee1b1b033"
server_key_iv: "5d072c1aff0ecf6217a5103172c523ac"
server_key_version: "encryptionkey@0"
I20260929 16:26:11.439781 3320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "9df102e2799d4b8d914e61b610da2f56"
format_stamp: "Formatted at 2026-09-29 16:26:11 on dist-test-slave-vq9v"
server_key: "f51950ec0744510410fcc05ee1b1b033"
server_key_iv: "5d072c1aff0ecf6217a5103172c523ac"
server_key_version: "encryptionkey@0"
I20260929 16:26:11.443876 3320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 16:26:11.446828 3335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:11.448261 3320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:26:11.448426 3320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "9df102e2799d4b8d914e61b610da2f56"
format_stamp: "Formatted at 2026-09-29 16:26:11 on dist-test-slave-vq9v"
server_key: "f51950ec0744510410fcc05ee1b1b033"
server_key_iv: "5d072c1aff0ecf6217a5103172c523ac"
server_key_version: "encryptionkey@0"
I20260929 16:26:11.448570 3320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:11.463987 3320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:11.467290 3320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:11.467520 3320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:11.474895 3320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33565
I20260929 16:26:11.475256 3387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33565 every 8 connection(s)
I20260929 16:26:11.475919 3320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:26:11.478920 3388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:11.479761 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 3320
I20260929 16:26:11.479900 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:26:11.480188 392 external_mini_cluster.cc:1428] Setting key df337ac62d6e7b2e3ad6ea74cb9b9a19
I20260929 16:26:11.485481 3388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56: Bootstrap starting.
I20260929 16:26:11.488257 3388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:11.489302 3388 log.cc:824] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:26:11 dist-test-slave-vq9v krb5kdc[3307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:11.491549 3388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56: No bootstrap required, opened a new log
I20260929 16:26:11.495237 3388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df102e2799d4b8d914e61b610da2f56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33565 } }
I20260929 16:26:11.495412 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:11.481564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49834 (local address 127.0.98.62:33565)
0929 16:26:11.482012 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:11.482029 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:11.482064 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:26:11.482706 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:11.482723 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:11.483059 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:11.483349 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:11.484308 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:11.485386 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:11.486589 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:11.487149 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:11.487282 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:11.490555 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:11.490561 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:11.490576 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:11.490606 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:11.492478 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:11.492982 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:11.492989 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:11.492993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:11.493114 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:11.493449 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:11.493453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:11.493457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:11.493860 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:11.494075 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:11.494363 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 16:26:11.494654 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":137,"threads_started":1}
I20260929 16:26:11.495467 3388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:11.495529 3388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df102e2799d4b8d914e61b610da2f56, State: Initialized, Role: FOLLOWER
I20260929 16:26:11.495963 3388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df102e2799d4b8d914e61b610da2f56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33565 } }
I20260929 16:26:11.496084 3388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:26:11.496167 3388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:26:11.496272 3388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:11.497283 3388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df102e2799d4b8d914e61b610da2f56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33565 } }
I20260929 16:26:11.497711 3388 leader_election.cc:302] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9df102e2799d4b8d914e61b610da2f56; no voters:
I20260929 16:26:11.498095 3388 leader_election.cc:288] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:26:11.498240 3393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:11.498317 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:26:11.498431 3393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56 [term 1 LEADER]: Becoming Leader. State: Replica: 9df102e2799d4b8d914e61b610da2f56, State: Running, Role: LEADER
I20260929 16:26:11.498723 3393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df102e2799d4b8d914e61b610da2f56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33565 } }
I20260929 16:26:11.499387 3388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:26:11.499908 3393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9df102e2799d4b8d914e61b610da2f56. Latest consensus state: current_term: 1 leader_uuid: "9df102e2799d4b8d914e61b610da2f56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df102e2799d4b8d914e61b610da2f56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33565 } } }
I20260929 16:26:11.500030 3393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:11.500347 3395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9df102e2799d4b8d914e61b610da2f56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df102e2799d4b8d914e61b610da2f56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33565 } } }
I20260929 16:26:11.500461 3395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:11.501487 3399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:26:11.504554 3399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:26:11.510223 3399 catalog_manager.cc:1347] Generated new cluster ID: 86f080f9f9474d13916ac88ea084dac4
I20260929 16:26:11.510315 3399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:26:11.522598 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:11.516796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49840 (local address 127.0.98.62:33565)
0929 16:26:11.516933 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:11.516938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:11.517301 (+ 363us) server_negotiation.cc:407] Connection header received
0929 16:26:11.517441 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:11.517444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:11.517495 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:11.517580 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:11.518193 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:11.518937 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:11.520130 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:11.520462 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:11.520555 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:11.521516 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:11.521520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:11.521522 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:11.522112 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:11.522145 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:11.522162 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:11.522243 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:11.522363 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 16:26:11.522447 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:26:11.532277 3399 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:26:11.533354 3399 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:26:11.542439 3399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9df102e2799d4b8d914e61b610da2f56: Generated new TSK 0
I20260929 16:26:11.543367 3399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:26:11.555737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 3320
2026-09-29T16:26:11Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1453 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:26:11 dist-test-slave-vq9v krb5kdc[3421](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:26:11 dist-test-slave-vq9v krb5kdc[3421](info): set up 2 sockets
Sep 29 16:26:11 dist-test-slave-vq9v krb5kdc[3421](info): commencing operation
krb5kdc: starting...
W20260929 16:26:12.633705 392 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260929 16:26:12.871443 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.191s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260929 16:26:13.074954 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.203s user 0.002s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:26:13 dist-test-slave-vq9v krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:26:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:26:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:26:13.145517 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32807
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:13.262233 3434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:13.262575 3434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:13.262657 3434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:13.267067 3434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:26:13.267177 3434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:13.267230 3434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:13.267289 3434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:26:13.267323 3434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:26:13.272912 3434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:13.274060 3434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:13.275085 3434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:13.283820 3439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:13.283816 3440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:13.283942 3434 server_base.cc:1034] running on GCE node
W20260929 16:26:13.283847 3442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:13.284664 3434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:13.285867 3434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:13.287124 3434 hybrid_clock.cc:648] HybridClock initialized: now 1790699173287094 us; error 46 us; skew 500 ppm
Sep 29 16:26:13 dist-test-slave-vq9v krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:13.294955 3434 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:26:13.296015 3434 webserver.cc:460] Webserver started at http://127.0.98.62:35517/ using document root <none> and password file <none>
I20260929 16:26:13.296617 3434 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:13.296710 3434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:13.296953 3434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:13.299304 3434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "bc10a14024e84c5e94e504d949cea98a"
format_stamp: "Formatted at 2026-09-29 16:26:13 on dist-test-slave-vq9v"
server_key: "cf2b34f612be28d5c734bb76ad97ae0b"
server_key_iv: "9de7996492224c5bf5f92991c974e1b2"
server_key_version: "encryptionkey@0"
I20260929 16:26:13.299894 3434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "bc10a14024e84c5e94e504d949cea98a"
format_stamp: "Formatted at 2026-09-29 16:26:13 on dist-test-slave-vq9v"
server_key: "cf2b34f612be28d5c734bb76ad97ae0b"
server_key_iv: "9de7996492224c5bf5f92991c974e1b2"
server_key_version: "encryptionkey@0"
I20260929 16:26:13.303916 3434 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:26:13.306727 3449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:13.308176 3434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:26:13.308354 3434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "bc10a14024e84c5e94e504d949cea98a"
format_stamp: "Formatted at 2026-09-29 16:26:13 on dist-test-slave-vq9v"
server_key: "cf2b34f612be28d5c734bb76ad97ae0b"
server_key_iv: "9de7996492224c5bf5f92991c974e1b2"
server_key_version: "encryptionkey@0"
I20260929 16:26:13.308504 3434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:13.338424 3434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:13.341734 3434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:13.341985 3434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:13.349318 3434 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32807
I20260929 16:26:13.349593 3501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32807 every 8 connection(s)
I20260929 16:26:13.350373 3434 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:26:13.352690 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 3434
I20260929 16:26:13.352792 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:26:13.353109 392 external_mini_cluster.cc:1428] Setting key e5011edc389402ffed1e915c87bd8421
I20260929 16:26:13.353463 3502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:13.360459 3502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a: Bootstrap starting.
Sep 29 16:26:13 dist-test-slave-vq9v krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:13.363300 3502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:13.364158 3502 log.cc:824] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:13.366214 3502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a: No bootstrap required, opened a new log
I20260929 16:26:13.368041 3505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:13.354581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.0.98.62:32807)
0929 16:26:13.354984 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:13.355000 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:13.355041 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:26:13.356101 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:13.356120 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:13.356443 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:13.356724 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:13.357737 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:13.358767 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:13.359958 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:13.360440 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:13.360598 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:13.363385 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:13.363394 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:13.363413 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:13.363450 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:13.365188 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:13.365696 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:13.365702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:13.365715 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:13.365821 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:13.366109 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:13.366114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:13.366115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:13.366557 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:13.366766 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:13.367037 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 16:26:13.367320 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":134,"threads_started":1}
I20260929 16:26:13.369937 3502 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc10a14024e84c5e94e504d949cea98a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32807 } }
I20260929 16:26:13.370222 3502 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:13.370337 3502 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc10a14024e84c5e94e504d949cea98a, State: Initialized, Role: FOLLOWER
I20260929 16:26:13.370743 3502 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc10a14024e84c5e94e504d949cea98a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32807 } }
I20260929 16:26:13.370863 3502 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:26:13.370901 3502 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:26:13.370972 3502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:13.371881 3502 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc10a14024e84c5e94e504d949cea98a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32807 } }
I20260929 16:26:13.372174 3502 leader_election.cc:302] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc10a14024e84c5e94e504d949cea98a; no voters:
I20260929 16:26:13.372447 3502 leader_election.cc:288] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:26:13.372655 3507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:13.372973 3507 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a [term 1 LEADER]: Becoming Leader. State: Replica: bc10a14024e84c5e94e504d949cea98a, State: Running, Role: LEADER
I20260929 16:26:13.373359 3507 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc10a14024e84c5e94e504d949cea98a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32807 } }
I20260929 16:26:13.373764 3502 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:26:13.375039 3508 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc10a14024e84c5e94e504d949cea98a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc10a14024e84c5e94e504d949cea98a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32807 } } }
I20260929 16:26:13.375170 3508 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:13.375574 3516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:26:13.375550 3509 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc10a14024e84c5e94e504d949cea98a. Latest consensus state: current_term: 1 leader_uuid: "bc10a14024e84c5e94e504d949cea98a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc10a14024e84c5e94e504d949cea98a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32807 } } }
I20260929 16:26:13.375695 3509 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:13.378934 3516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:26:13.384747 3516 catalog_manager.cc:1347] Generated new cluster ID: 33caa6c191534b1d83f90048cff90710
I20260929 16:26:13.384871 3516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:26:13.393046 3516 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:26:13.394188 3516 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:26:13.404601 3516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc10a14024e84c5e94e504d949cea98a: Generated new TSK 0
I20260929 16:26:13.405560 3516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 16:26:13.444626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32807
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:13.560099 3528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:13.560441 3528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:13.560528 3528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:13.564778 3528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:13.564886 3528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:13.564978 3528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:26:13.570513 3528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:13.571642 3528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:13.572707 3528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:26:13.583180 3528 server_base.cc:1034] running on GCE node
W20260929 16:26:13.583168 3534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:13.583168 3536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:13.583381 3533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:13.583822 3528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:13.584513 3528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:13.585747 3528 hybrid_clock.cc:648] HybridClock initialized: now 1790699173585730 us; error 32 us; skew 500 ppm
Sep 29 16:26:13 dist-test-slave-vq9v krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:13.589052 3528 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:26:13.590109 3528 webserver.cc:460] Webserver started at http://127.0.98.1:46289/ using document root <none> and password file <none>
I20260929 16:26:13.590711 3528 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:13.590811 3528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:13.591053 3528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:13.593613 3528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "ca472596bff74986830c37deacb039de"
format_stamp: "Formatted at 2026-09-29 16:26:13 on dist-test-slave-vq9v"
server_key: "58f18ea868aa9f2df0f9e9884c7e3d99"
server_key_iv: "d1b1dd6653868618576ac60eda5d6487"
server_key_version: "encryptionkey@0"
I20260929 16:26:13.594205 3528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ca472596bff74986830c37deacb039de"
format_stamp: "Formatted at 2026-09-29 16:26:13 on dist-test-slave-vq9v"
server_key: "58f18ea868aa9f2df0f9e9884c7e3d99"
server_key_iv: "d1b1dd6653868618576ac60eda5d6487"
server_key_version: "encryptionkey@0"
I20260929 16:26:13.598246 3528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:26:13.601317 3543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:13.602828 3528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:13.603010 3528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "ca472596bff74986830c37deacb039de"
format_stamp: "Formatted at 2026-09-29 16:26:13 on dist-test-slave-vq9v"
server_key: "58f18ea868aa9f2df0f9e9884c7e3d99"
server_key_iv: "d1b1dd6653868618576ac60eda5d6487"
server_key_version: "encryptionkey@0"
I20260929 16:26:13.603152 3528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:13.620901 3528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:13.624521 3528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:13.624745 3528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:13.625437 3528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:13.626299 3528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:13.626372 3528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:13.626444 3528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:13.626499 3528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:13.635825 3528 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37897
I20260929 16:26:13.635849 3656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37897 every 8 connection(s)
I20260929 16:26:13.636750 3528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:26:13.641882 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 3528
I20260929 16:26:13.641990 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:26:13.642338 392 external_mini_cluster.cc:1428] Setting key 72dba4824280b507dad3c3a2665417b3
Sep 29 16:26:13 dist-test-slave-vq9v krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 16:26:13.653568 3505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:13.638767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37027 (local address 127.0.98.62:32807)
0929 16:26:13.638943 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:13.638947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:13.639768 (+ 821us) server_negotiation.cc:407] Connection header received
0929 16:26:13.641414 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:13.641418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:13.641481 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:13.641610 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:13.643211 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:13.643803 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:13.645134 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:13.645485 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:13.645640 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:13.648975 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:13.648982 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:13.648989 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:13.649039 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:13.650822 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:13.651494 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:13.651500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:13.651502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:13.651598 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:13.652082 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:13.652088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:13.652091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:13.652381 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:13.652496 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:13.653307 (+ 811us) server_negotiation.cc:299] Negotiation successful
0929 16:26:13.653435 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:26:13.654534 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:13.639120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32807 (local address 127.0.98.1:37027)
0929 16:26:13.639602 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:13.639648 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:13.641122 (+ 1474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:13.641822 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:13.641829 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:13.642348 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:13.643045 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:13.643063 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:13.643949 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:13.643959 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:13.644966 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:13.644976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:13.645704 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:13.645709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:13.645904 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:13.646713 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:13.646748 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:13.648727 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:13.650991 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:13.650997 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:13.651014 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:13.651365 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:13.651749 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:13.651752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:13.651754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:13.651921 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:13.652531 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:13.652537 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:13.652898 (+ 361us) client_negotiation.cc:769] Sending connection context
0929 16:26:13.653149 (+ 251us) client_negotiation.cc:240] Negotiation successful
0929 16:26:13.653534 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":128,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 16:26:13.656165 3657 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32807
I20260929 16:26:13.656445 3657 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:13.656983 3657 heartbeater.cc:511] Master 127.0.98.62:32807 requested a full tablet report, sending...
I20260929 16:26:13.658763 3466 ts_manager.cc:194] Registered new tserver with Master: ca472596bff74986830c37deacb039de (127.0.98.1:37897)
I20260929 16:26:13.660092 3466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 16:26:13.673808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32807
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:13.794886 3662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:13.795195 3662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:13.795284 3662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:13.799477 3662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:13.799579 3662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:13.799682 3662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:26:13.805188 3662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:13.806470 3662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:13.807616 3662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:13.818758 3670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:13.818750 3667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:13.818805 3662 server_base.cc:1034] running on GCE node
W20260929 16:26:13.818748 3668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:13.819546 3662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:13.820258 3662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:13.821451 3662 hybrid_clock.cc:648] HybridClock initialized: now 1790699173821425 us; error 45 us; skew 500 ppm
Sep 29 16:26:13 dist-test-slave-vq9v krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:13.824517 3662 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:26:13.825595 3662 webserver.cc:460] Webserver started at http://127.0.98.2:35717/ using document root <none> and password file <none>
I20260929 16:26:13.826174 3662 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:13.826267 3662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:13.826500 3662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:13.828634 3662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "78581887a4bd4dd9b3418e647e49a9ea"
format_stamp: "Formatted at 2026-09-29 16:26:13 on dist-test-slave-vq9v"
server_key: "8f5f5fc3c6f9f0388e14b31014c79957"
server_key_iv: "01f7bdeab194fe65235daeeceb1d0dd4"
server_key_version: "encryptionkey@0"
I20260929 16:26:13.829257 3662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "78581887a4bd4dd9b3418e647e49a9ea"
format_stamp: "Formatted at 2026-09-29 16:26:13 on dist-test-slave-vq9v"
server_key: "8f5f5fc3c6f9f0388e14b31014c79957"
server_key_iv: "01f7bdeab194fe65235daeeceb1d0dd4"
server_key_version: "encryptionkey@0"
I20260929 16:26:13.833081 3662 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 16:26:13.835757 3677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:13.837312 3662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:13.837458 3662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "78581887a4bd4dd9b3418e647e49a9ea"
format_stamp: "Formatted at 2026-09-29 16:26:13 on dist-test-slave-vq9v"
server_key: "8f5f5fc3c6f9f0388e14b31014c79957"
server_key_iv: "01f7bdeab194fe65235daeeceb1d0dd4"
server_key_version: "encryptionkey@0"
I20260929 16:26:13.837590 3662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:13.865836 3662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:13.869450 3662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:13.869674 3662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:13.870275 3662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:13.871127 3662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:13.871198 3662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:13.871268 3662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:13.871322 3662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:13.881399 3662 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35029
I20260929 16:26:13.881454 3790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35029 every 8 connection(s)
I20260929 16:26:13.882432 3662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:26:13.891744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 3662
I20260929 16:26:13.891885 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:26:13.892202 392 external_mini_cluster.cc:1428] Setting key a57575e9ecd3da12a43e993a3eedb37d
Sep 29 16:26:13 dist-test-slave-vq9v krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:13.898248 3505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:13.884347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42001 (local address 127.0.98.62:32807)
0929 16:26:13.884523 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:13.884527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:13.885429 (+ 902us) server_negotiation.cc:407] Connection header received
0929 16:26:13.887018 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:13.887022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:13.887086 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:13.887201 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:13.888517 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:13.889136 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:13.890524 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:13.890779 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:13.890906 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:13.894032 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:13.894035 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:13.894037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:13.894061 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:13.895820 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:13.896384 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:13.896391 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:13.896395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:13.896485 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:13.896820 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:13.896826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:13.896831 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:13.897097 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:13.897236 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:13.897891 (+ 655us) server_negotiation.cc:299] Negotiation successful
0929 16:26:13.898051 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:26:13.899044 3793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:13.884665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32807 (local address 127.0.98.2:42001)
0929 16:26:13.885163 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:13.885288 (+ 125us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:13.886774 (+ 1486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:13.887400 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:13.887408 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:13.887818 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:13.888338 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:13.888351 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:13.889376 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:13.889380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:13.890375 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:13.890384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:13.890947 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:13.890952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:13.891139 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:13.891980 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:13.892010 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:13.893820 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:13.895965 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:13.895971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:13.895984 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:13.896275 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:13.896590 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:13.896592 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:13.896594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:13.896734 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:13.897271 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:13.897284 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:13.897554 (+ 270us) client_negotiation.cc:769] Sending connection context
0929 16:26:13.897815 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 16:26:13.898123 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":122,"threads_started":1}
I20260929 16:26:13.900334 3791 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32807
I20260929 16:26:13.900578 3791 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:13.901078 3791 heartbeater.cc:511] Master 127.0.98.62:32807 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 16:26:13.902257 3466 ts_manager.cc:194] Registered new tserver with Master: 78581887a4bd4dd9b3418e647e49a9ea (127.0.98.2:35029)
I20260929 16:26:13.902873 3466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 16:26:13.925367 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32807
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:14.038549 3796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:14.038832 3796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:14.038877 3796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:14.043088 3796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:14.043159 3796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:14.043220 3796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:26:14.048504 3796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:14.049571 3796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:14.050542 3796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:14.060755 3801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:14.060792 3796 server_base.cc:1034] running on GCE node
W20260929 16:26:14.060730 3802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:14.060714 3804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:14.061455 3796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:14.062242 3796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:14.063448 3796 hybrid_clock.cc:648] HybridClock initialized: now 1790699174063433 us; error 26 us; skew 500 ppm
Sep 29 16:26:14 dist-test-slave-vq9v krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:14.066581 3796 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:26:14.067483 3796 webserver.cc:460] Webserver started at http://127.0.98.3:46613/ using document root <none> and password file <none>
I20260929 16:26:14.068025 3796 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:14.068086 3796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:14.068252 3796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:14.070418 3796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "f1871aa5cf95417c9ebaaa4a85ee0926"
format_stamp: "Formatted at 2026-09-29 16:26:14 on dist-test-slave-vq9v"
server_key: "4dc088e1b2a5c9a8474eb0ede465c0bc"
server_key_iv: "b1c621ca55ea406291b69e043cf6a6f7"
server_key_version: "encryptionkey@0"
I20260929 16:26:14.070986 3796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f1871aa5cf95417c9ebaaa4a85ee0926"
format_stamp: "Formatted at 2026-09-29 16:26:14 on dist-test-slave-vq9v"
server_key: "4dc088e1b2a5c9a8474eb0ede465c0bc"
server_key_iv: "b1c621ca55ea406291b69e043cf6a6f7"
server_key_version: "encryptionkey@0"
I20260929 16:26:14.075350 3796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:26:14.078142 3811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:14.079694 3796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:14.079815 3796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "f1871aa5cf95417c9ebaaa4a85ee0926"
format_stamp: "Formatted at 2026-09-29 16:26:14 on dist-test-slave-vq9v"
server_key: "4dc088e1b2a5c9a8474eb0ede465c0bc"
server_key_iv: "b1c621ca55ea406291b69e043cf6a6f7"
server_key_version: "encryptionkey@0"
I20260929 16:26:14.080078 3796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:14.109622 3796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:14.112943 3796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:14.113137 3796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:14.113857 3796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:14.114705 3796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:14.114750 3796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:14.114815 3796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:14.114847 3796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:14.124356 3796 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35579
I20260929 16:26:14.124384 3924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35579 every 8 connection(s)
I20260929 16:26:14.125393 3796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:26:14.132588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 3796
I20260929 16:26:14.132732 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:26:14.133033 392 external_mini_cluster.cc:1428] Setting key 67eaa2cb988fe3826d649ac7ce4fea96
Sep 29 16:26:14 dist-test-slave-vq9v krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:14.141170 3505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:14.127262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56675 (local address 127.0.98.62:32807)
0929 16:26:14.127448 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:14.127452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:14.128556 (+ 1104us) server_negotiation.cc:407] Connection header received
0929 16:26:14.130133 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:14.130136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:14.130190 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:14.130272 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:14.131597 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:14.132246 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:14.133889 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:14.134255 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:14.134347 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:14.137183 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:14.137187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:14.137189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:14.137284 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:14.138829 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:14.139383 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:14.139386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:14.139387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:14.139485 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:14.139842 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:14.139845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:14.139846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:14.140095 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:14.140195 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:14.140824 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 16:26:14.140961 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:26:14.142117 3927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:14.127659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32807 (local address 127.0.98.3:56675)
0929 16:26:14.128372 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:14.128420 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:14.129820 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:14.130464 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:14.130471 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:14.130899 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:14.131416 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:14.131431 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:14.132429 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:14.132432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:14.133773 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:14.133783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:14.134441 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:14.134445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:14.134604 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:14.135269 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:14.135287 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:14.136995 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:14.138956 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:14.138962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:14.138976 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:14.139283 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:14.139586 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:14.139588 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:14.139589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:14.139725 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:14.140264 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:14.140271 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:14.140563 (+ 292us) client_negotiation.cc:769] Sending connection context
0929 16:26:14.140738 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 16:26:14.141124 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":309,"threads_started":1}
I20260929 16:26:14.143469 3925 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32807
I20260929 16:26:14.143730 3925 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:14.144284 3925 heartbeater.cc:511] Master 127.0.98.62:32807 requested a full tablet report, sending...
I20260929 16:26:14.145498 3466 ts_manager.cc:194] Registered new tserver with Master: f1871aa5cf95417c9ebaaa4a85ee0926 (127.0.98.3:35579)
I20260929 16:26:14.146198 3466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56675
I20260929 16:26:14.147679 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:26:14 dist-test-slave-vq9v krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699174, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 16:26:14 dist-test-slave-vq9v krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699174, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:14.174888 3505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:14.165973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.0.98.62:32807)
0929 16:26:14.166216 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:14.166220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:14.166242 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:26:14.166385 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:14.166387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:14.166434 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:14.166515 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:14.167346 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:14.167891 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:14.168813 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:14.169037 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:14.169125 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:14.171659 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:14.171663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:14.171665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:14.171702 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:14.173335 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:14.173729 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:14.173732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:14.173733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:14.173810 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:14.174025 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:14.174027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:14.174028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:14.174245 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:14.174347 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:14.174529 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 16:26:14.174626 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260929 16:26:14.175387 3466 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41806
I20260929 16:26:14.176323 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 3528
I20260929 16:26:14.181778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 3662
I20260929 16:26:14.186903 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 3796
I20260929 16:26:14.192380 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 3434
2026-09-29T16:26:14Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2642 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:26:14 dist-test-slave-vq9v krb5kdc[3937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:26:14 dist-test-slave-vq9v krb5kdc[3937](info): set up 2 sockets
Sep 29 16:26:14 dist-test-slave-vq9v krb5kdc[3937](info): commencing operation
krb5kdc: starting...
W20260929 16:26:15.257306 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:26:15 dist-test-slave-vq9v krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:26:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:26:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:26:15.385547 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38595
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:15.498370 3950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:15.498689 3950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:15.498791 3950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:15.502890 3950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:26:15.502980 3950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:15.503019 3950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:15.503084 3950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:26:15.503123 3950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:26:15.508524 3950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:15.509778 3950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:15.510845 3950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:15.519625 3958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:15.519658 3956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:15.519625 3955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:15.519733 3950 server_base.cc:1034] running on GCE node
I20260929 16:26:15.520625 3950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:15.521863 3950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:15.523093 3950 hybrid_clock.cc:648] HybridClock initialized: now 1790699175523064 us; error 44 us; skew 500 ppm
Sep 29 16:26:15 dist-test-slave-vq9v krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:15.525909 3950 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:26:15.526851 3950 webserver.cc:460] Webserver started at http://127.0.98.62:40307/ using document root <none> and password file <none>
I20260929 16:26:15.527431 3950 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:15.527527 3950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:15.527766 3950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:15.529917 3950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "8f8f2411b47341d8bce0fde0b79a4523"
format_stamp: "Formatted at 2026-09-29 16:26:15 on dist-test-slave-vq9v"
server_key: "c67ed94fe7d3918fadac680e20273a0c"
server_key_iv: "50a46d880f5e3bad6dc77bf368c84fc7"
server_key_version: "encryptionkey@0"
I20260929 16:26:15.530493 3950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "8f8f2411b47341d8bce0fde0b79a4523"
format_stamp: "Formatted at 2026-09-29 16:26:15 on dist-test-slave-vq9v"
server_key: "c67ed94fe7d3918fadac680e20273a0c"
server_key_iv: "50a46d880f5e3bad6dc77bf368c84fc7"
server_key_version: "encryptionkey@0"
I20260929 16:26:15.534502 3950 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:26:15.536931 3965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:15.538273 3950 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:26:15.538410 3950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "8f8f2411b47341d8bce0fde0b79a4523"
format_stamp: "Formatted at 2026-09-29 16:26:15 on dist-test-slave-vq9v"
server_key: "c67ed94fe7d3918fadac680e20273a0c"
server_key_iv: "50a46d880f5e3bad6dc77bf368c84fc7"
server_key_version: "encryptionkey@0"
I20260929 16:26:15.538527 3950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:15.550916 3950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:15.553911 3950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:15.554106 3950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:15.561918 3950 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38595
I20260929 16:26:15.562093 4017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38595 every 8 connection(s)
I20260929 16:26:15.562964 3950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:26:15.565724 4018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:15.571210 4018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523: Bootstrap starting.
I20260929 16:26:15.572228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 3950
I20260929 16:26:15.572346 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:26:15.572678 392 external_mini_cluster.cc:1428] Setting key ec54f365cdf9bba5878642240a0d1026
I20260929 16:26:15.574328 4018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:15.575141 4018 log.cc:824] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:15.577615 4018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523: No bootstrap required, opened a new log
Sep 29 16:26:15 dist-test-slave-vq9v krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:15.582609 4018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8f2411b47341d8bce0fde0b79a4523" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38595 } }
I20260929 16:26:15.582876 4018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:15.582937 4018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f8f2411b47341d8bce0fde0b79a4523, State: Initialized, Role: FOLLOWER
I20260929 16:26:15.583463 4018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8f2411b47341d8bce0fde0b79a4523" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38595 } }
I20260929 16:26:15.583636 4018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:26:15.583732 4018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:26:15.583848 4018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:15.585131 4018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8f2411b47341d8bce0fde0b79a4523" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38595 } }
I20260929 16:26:15.585554 4018 leader_election.cc:302] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f8f2411b47341d8bce0fde0b79a4523; no voters:
I20260929 16:26:15.585906 4018 leader_election.cc:288] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:26:15.586063 4023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:15.586304 4023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523 [term 1 LEADER]: Becoming Leader. State: Replica: 8f8f2411b47341d8bce0fde0b79a4523, State: Running, Role: LEADER
I20260929 16:26:15.586669 4023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8f2411b47341d8bce0fde0b79a4523" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38595 } }
I20260929 16:26:15.587337 4018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:26:15.588311 4024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f8f2411b47341d8bce0fde0b79a4523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8f2411b47341d8bce0fde0b79a4523" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38595 } } }
I20260929 16:26:15.588425 4024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:15.588737 4025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f8f2411b47341d8bce0fde0b79a4523. Latest consensus state: current_term: 1 leader_uuid: "8f8f2411b47341d8bce0fde0b79a4523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8f2411b47341d8bce0fde0b79a4523" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38595 } } }
I20260929 16:26:15.588811 4025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:15.588784 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:15.573805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39264 (local address 127.0.98.62:38595)
0929 16:26:15.574473 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:15.574494 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:15.574538 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:26:15.575177 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:15.575193 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:15.575655 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:15.576023 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:15.576927 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:15.578054 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:15.579232 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:15.579740 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:15.579880 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:15.583071 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:15.583077 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:15.583092 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:15.583123 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:15.585328 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:15.585810 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:15.585817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:15.585827 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:15.585984 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:15.586329 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:15.586332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:15.586334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:15.586835 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:15.587036 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:15.587342 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 16:26:15.587698 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":130,"threads_started":1}
I20260929 16:26:15.589944 4032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:26:15.592440 4032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:26:15.597640 4032 catalog_manager.cc:1347] Generated new cluster ID: 9996e952e25849c2a1178a57ff3ae01a
I20260929 16:26:15.597712 4032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:26:15.605494 4032 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:26:15.606806 4032 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:26:15.615155 4032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f8f2411b47341d8bce0fde0b79a4523: Generated new TSK 0
I20260929 16:26:15.615890 4032 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 16:26:15.649005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38595
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:15.761672 4044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:15.761977 4044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:15.762025 4044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:15.766265 4044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:15.766340 4044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:15.766429 4044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:26:15.771862 4044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:15.772976 4044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:15.774080 4044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:26:15.784042 4044 server_base.cc:1034] running on GCE node
W20260929 16:26:15.784057 4052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:15.783962 4049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:15.783974 4050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:15.784629 4044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:15.785356 4044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:15.786564 4044 hybrid_clock.cc:648] HybridClock initialized: now 1790699175786542 us; error 35 us; skew 500 ppm
Sep 29 16:26:15 dist-test-slave-vq9v krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:15.789686 4044 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:26:15.790558 4044 webserver.cc:460] Webserver started at http://127.0.98.1:38931/ using document root <none> and password file <none>
I20260929 16:26:15.791090 4044 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:15.791149 4044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:15.791311 4044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:15.793506 4044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "3309a1a77323496b9a9aa052339a8348"
format_stamp: "Formatted at 2026-09-29 16:26:15 on dist-test-slave-vq9v"
server_key: "71190332b9eba7b6ae20371732759c98"
server_key_iv: "3a9dfd6e0d358c092ee38efb5bee65f8"
server_key_version: "encryptionkey@0"
I20260929 16:26:15.794315 4044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3309a1a77323496b9a9aa052339a8348"
format_stamp: "Formatted at 2026-09-29 16:26:15 on dist-test-slave-vq9v"
server_key: "71190332b9eba7b6ae20371732759c98"
server_key_iv: "3a9dfd6e0d358c092ee38efb5bee65f8"
server_key_version: "encryptionkey@0"
I20260929 16:26:15.798646 4044 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 16:26:15.801386 4059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:15.802649 4044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:26:15.802791 4044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "3309a1a77323496b9a9aa052339a8348"
format_stamp: "Formatted at 2026-09-29 16:26:15 on dist-test-slave-vq9v"
server_key: "71190332b9eba7b6ae20371732759c98"
server_key_iv: "3a9dfd6e0d358c092ee38efb5bee65f8"
server_key_version: "encryptionkey@0"
I20260929 16:26:15.802912 4044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:15.814517 4044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:15.817951 4044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:15.818168 4044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:15.818778 4044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:15.819614 4044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:15.819661 4044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:15.819728 4044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:15.819761 4044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:15.829017 4044 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45187
I20260929 16:26:15.829110 4172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45187 every 8 connection(s)
I20260929 16:26:15.829996 4044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:26:15.835935 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 4044
I20260929 16:26:15.836087 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:26:15.836387 392 external_mini_cluster.cc:1428] Setting key 5b33291893c18d9c840a1d3d185fb6b2
Sep 29 16:26:15 dist-test-slave-vq9v krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:15.846403 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:15.832000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43601 (local address 127.0.98.62:38595)
0929 16:26:15.832161 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:15.832165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:15.833059 (+ 894us) server_negotiation.cc:407] Connection header received
0929 16:26:15.834563 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:15.834567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:15.834631 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:15.834708 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:15.836142 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:15.836796 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:15.838375 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:15.838602 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:15.838687 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:15.842130 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:15.842134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:15.842136 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:15.842173 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:15.844017 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:15.844694 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:15.844696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:15.844698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:15.844772 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:15.845176 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:15.845179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:15.845180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:15.845423 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:15.845502 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:15.846170 (+ 668us) server_negotiation.cc:299] Negotiation successful
0929 16:26:15.846278 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 16:26:15.847605 4175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:15.832430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38595 (local address 127.0.98.1:43601)
0929 16:26:15.832897 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:15.832941 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:15.834337 (+ 1396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:15.834894 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:15.834901 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:15.835304 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:15.835900 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:15.835922 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:15.836971 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:15.836974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:15.838237 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:15.838249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:15.838761 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:15.838767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:15.839019 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:15.839734 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:15.839761 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:15.841925 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:15.844156 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:15.844165 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:15.844186 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:15.844585 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:15.844865 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:15.844868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:15.844871 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:15.845066 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:15.845531 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:15.845541 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:15.845897 (+ 356us) client_negotiation.cc:769] Sending connection context
0929 16:26:15.846136 (+ 239us) client_negotiation.cc:240] Negotiation successful
0929 16:26:15.846533 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":122,"threads_started":1}
I20260929 16:26:15.848884 4173 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38595
I20260929 16:26:15.849251 4173 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 16:26:15.849869 4173 heartbeater.cc:511] Master 127.0.98.62:38595 requested a full tablet report, sending...
I20260929 16:26:15.851522 3979 ts_manager.cc:194] Registered new tserver with Master: 3309a1a77323496b9a9aa052339a8348 (127.0.98.1:45187)
I20260929 16:26:15.852726 3979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 16:26:15.867925 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38595
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:15.982287 4178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:15.982577 4178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:15.982649 4178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:15.987067 4178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:15.987170 4178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:15.987264 4178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:26:15.993048 4178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:15.994203 4178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:15.995277 4178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:26:16.005708 4178 server_base.cc:1034] running on GCE node
W20260929 16:26:16.005585 4183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:16.005585 4184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:16.005628 4186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:16.006381 4178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:16.007200 4178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:16.008410 4178 hybrid_clock.cc:648] HybridClock initialized: now 1790699176008397 us; error 30 us; skew 500 ppm
Sep 29 16:26:16 dist-test-slave-vq9v krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:16.011577 4178 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:26:16.012521 4178 webserver.cc:460] Webserver started at http://127.0.98.2:37663/ using document root <none> and password file <none>
I20260929 16:26:16.013108 4178 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:16.013202 4178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:16.013475 4178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:16.015658 4178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "736af2974ab546888cf2b585600f1939"
format_stamp: "Formatted at 2026-09-29 16:26:16 on dist-test-slave-vq9v"
server_key: "0e2c6dd6d7d63f808c68c157e725fce2"
server_key_iv: "c27beb8c3dc7fc9073e890dbcf1ca1ce"
server_key_version: "encryptionkey@0"
I20260929 16:26:16.016233 4178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "736af2974ab546888cf2b585600f1939"
format_stamp: "Formatted at 2026-09-29 16:26:16 on dist-test-slave-vq9v"
server_key: "0e2c6dd6d7d63f808c68c157e725fce2"
server_key_iv: "c27beb8c3dc7fc9073e890dbcf1ca1ce"
server_key_version: "encryptionkey@0"
I20260929 16:26:16.020329 4178 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:26:16.023002 4193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:16.024371 4178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:26:16.024529 4178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "736af2974ab546888cf2b585600f1939"
format_stamp: "Formatted at 2026-09-29 16:26:16 on dist-test-slave-vq9v"
server_key: "0e2c6dd6d7d63f808c68c157e725fce2"
server_key_iv: "c27beb8c3dc7fc9073e890dbcf1ca1ce"
server_key_version: "encryptionkey@0"
I20260929 16:26:16.024681 4178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:16.036164 4178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:16.039629 4178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:16.039943 4178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:16.040573 4178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:16.041518 4178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:16.041599 4178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:16.041694 4178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:16.041747 4178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:16.051986 4178 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44445
I20260929 16:26:16.052016 4306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44445 every 8 connection(s)
I20260929 16:26:16.052974 4178 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:26:16.054566 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 4178
I20260929 16:26:16.054739 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:26:16.055035 392 external_mini_cluster.cc:1428] Setting key 240647fcfdfc15aaa642eb7dcd0fd6c8
Sep 29 16:26:16 dist-test-slave-vq9v krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 16:26:16.070281 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:16.055311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48561 (local address 127.0.98.62:38595)
0929 16:26:16.055481 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:16.055484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:16.056591 (+ 1107us) server_negotiation.cc:407] Connection header received
0929 16:26:16.058795 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:16.058801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:16.058869 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:16.058990 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:16.060510 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:16.061042 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:16.062322 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:16.062562 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:16.062704 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:16.065653 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:16.065656 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:16.065659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:16.065683 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:16.067259 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:16.067889 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:16.067915 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:16.067921 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:16.068042 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:16.068562 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:16.068566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:16.068569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:16.068852 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:16.068966 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:16.069920 (+ 954us) server_negotiation.cc:299] Negotiation successful
0929 16:26:16.070092 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:26:16.071430 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:16.055826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38595 (local address 127.0.98.2:48561)
0929 16:26:16.056415 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:16.056477 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:16.058469 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:16.059184 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:16.059192 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:16.059565 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:16.060312 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:16.060336 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:16.061259 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:16.061263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:16.062176 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:16.062186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:16.062751 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:16.062755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:16.062938 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:16.063624 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:16.063642 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:16.065450 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:16.067398 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:16.067403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:16.067419 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:16.067726 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:16.068191 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:16.068198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:16.068203 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:16.068416 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:16.069000 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:16.069029 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:16.069465 (+ 436us) client_negotiation.cc:769] Sending connection context
0929 16:26:16.069793 (+ 328us) client_negotiation.cc:240] Negotiation successful
0929 16:26:16.070250 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":141,"threads_started":1}
I20260929 16:26:16.072855 4307 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38595
I20260929 16:26:16.073130 4307 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:16.073839 4307 heartbeater.cc:511] Master 127.0.98.62:38595 requested a full tablet report, sending...
I20260929 16:26:16.075143 3979 ts_manager.cc:194] Registered new tserver with Master: 736af2974ab546888cf2b585600f1939 (127.0.98.2:44445)
I20260929 16:26:16.076025 3979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 16:26:16.089023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38595
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:16.210079 4312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:16.210376 4312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:16.210426 4312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:16.216040 4312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:16.216138 4312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:16.216215 4312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:26:16.222111 4312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:16.223203 4312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:16.224179 4312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:16.234002 4317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:16.233990 4318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:16.234133 4312 server_base.cc:1034] running on GCE node
W20260929 16:26:16.233990 4320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:16.234737 4312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:16.235491 4312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:16.236752 4312 hybrid_clock.cc:648] HybridClock initialized: now 1790699176236715 us; error 51 us; skew 500 ppm
Sep 29 16:26:16 dist-test-slave-vq9v krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:16.239925 4312 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:26:16.240929 4312 webserver.cc:460] Webserver started at http://127.0.98.3:44809/ using document root <none> and password file <none>
I20260929 16:26:16.241586 4312 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:16.241680 4312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:16.241964 4312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:16.244145 4312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "e73362b80ac04b46bde63c636bd4a2c3"
format_stamp: "Formatted at 2026-09-29 16:26:16 on dist-test-slave-vq9v"
server_key: "737f5e61b1e70a80a0d0b64875b9b37d"
server_key_iv: "08e45f8144ce5dd9b7ea1d42d2b78ab7"
server_key_version: "encryptionkey@0"
I20260929 16:26:16.244735 4312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e73362b80ac04b46bde63c636bd4a2c3"
format_stamp: "Formatted at 2026-09-29 16:26:16 on dist-test-slave-vq9v"
server_key: "737f5e61b1e70a80a0d0b64875b9b37d"
server_key_iv: "08e45f8144ce5dd9b7ea1d42d2b78ab7"
server_key_version: "encryptionkey@0"
I20260929 16:26:16.248826 4312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:26:16.251509 4327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:16.252885 4312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:26:16.253099 4312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "e73362b80ac04b46bde63c636bd4a2c3"
format_stamp: "Formatted at 2026-09-29 16:26:16 on dist-test-slave-vq9v"
server_key: "737f5e61b1e70a80a0d0b64875b9b37d"
server_key_iv: "08e45f8144ce5dd9b7ea1d42d2b78ab7"
server_key_version: "encryptionkey@0"
I20260929 16:26:16.253288 4312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:16.263692 4312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:16.267369 4312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:16.267604 4312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:16.268239 4312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:16.269089 4312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:16.269164 4312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:16.269263 4312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:16.269309 4312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:16.278584 4312 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44823
I20260929 16:26:16.278604 4440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44823 every 8 connection(s)
I20260929 16:26:16.279523 4312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:26:16.285841 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 4312
I20260929 16:26:16.286001 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:26:16.286414 392 external_mini_cluster.cc:1428] Setting key 5955744b9bcd20aa8afa9c625f939957
Sep 29 16:26:16 dist-test-slave-vq9v krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:16.294760 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:16.281617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39153 (local address 127.0.98.62:38595)
0929 16:26:16.281762 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:16.281765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:16.282649 (+ 884us) server_negotiation.cc:407] Connection header received
0929 16:26:16.284086 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:16.284090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:16.284143 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:16.284230 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:16.285506 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:16.286160 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:16.287865 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:16.288179 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:16.288307 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:16.291077 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:16.291080 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:16.291083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:16.291107 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:16.292657 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:16.293285 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:16.293288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:16.293289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:16.293373 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:16.293714 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:16.293716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:16.293717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:16.293898 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:16.293975 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:16.294511 (+ 536us) server_negotiation.cc:299] Negotiation successful
0929 16:26:16.294638 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:26:16.295625 4443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:16.282056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38595 (local address 127.0.98.3:39153)
0929 16:26:16.282497 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:16.282542 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:16.283863 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:16.284419 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:16.284427 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:16.284795 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:16.285364 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:16.285380 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:16.286280 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:16.286283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:16.287757 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:16.287769 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:16.288385 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:16.288392 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:16.288597 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:16.289278 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:16.289305 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:16.290886 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:16.292778 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:16.292783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:16.292797 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:16.293088 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:16.293465 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:16.293468 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:16.293469 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:16.293627 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:16.293987 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:16.293994 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:16.294252 (+ 258us) client_negotiation.cc:769] Sending connection context
0929 16:26:16.294452 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 16:26:16.294742 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":128,"threads_started":1}
I20260929 16:26:16.296868 4441 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38595
I20260929 16:26:16.297137 4441 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:16.297683 4441 heartbeater.cc:511] Master 127.0.98.62:38595 requested a full tablet report, sending...
I20260929 16:26:16.298728 3979 ts_manager.cc:194] Registered new tserver with Master: e73362b80ac04b46bde63c636bd4a2c3 (127.0.98.3:44823)
I20260929 16:26:16.299422 3979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39153
I20260929 16:26:16.301023 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:26:16 dist-test-slave-vq9v krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:26:16 dist-test-slave-vq9v krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 16:26:16.327546 4448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:16.317609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37320 (local address 127.0.98.1:45187)
0929 16:26:16.317932 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:16.317956 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:16.317983 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:26:16.318061 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:16.318066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:16.318256 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:16.318392 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:16.319099 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:16.319919 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:16.320793 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:16.321284 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:16.321375 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:16.323852 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:16.323861 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:16.323875 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:16.323901 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:16.325772 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:16.326215 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:16.326220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:16.326222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:16.326305 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:16.326538 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:16.326540 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:16.326541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:16.326793 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:16.326963 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:16.327100 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 16:26:16.327324 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":105,"threads_started":1}
Sep 29 16:26:16 dist-test-slave-vq9v krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:16.338891 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:16.330641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38952 (local address 127.0.98.62:38595)
0929 16:26:16.330783 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:16.330788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:16.330854 (+ 66us) server_negotiation.cc:407] Connection header received
0929 16:26:16.331014 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:16.331022 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:16.331062 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:16.331153 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:16.331773 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:16.332298 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:16.333101 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:16.333396 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:16.333508 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:16.335848 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:16.335851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:16.335853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:16.335877 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:16.337452 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:16.337839 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:16.337841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:16.337842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:16.337917 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:16.338147 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:16.338149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:16.338150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:16.338360 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:16.338460 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:16.338673 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 16:26:16.338767 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 16:26:16.339341 3980 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38952
I20260929 16:26:16.339947 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 4044
I20260929 16:26:16.345939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 4178
I20260929 16:26:16.351604 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 4312
I20260929 16:26:16.356966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 3950
2026-09-29T16:26:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2162 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:26:16 dist-test-slave-vq9v krb5kdc[4453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:26:16 dist-test-slave-vq9v krb5kdc[4453](info): set up 2 sockets
Sep 29 16:26:16 dist-test-slave-vq9v krb5kdc[4453](info): commencing operation
krb5kdc: starting...
W20260929 16:26:17.409576 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:26:17 dist-test-slave-vq9v krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:26:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:26:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:26:17.536772 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43683
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:17.654436 4466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:17.654719 4466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:17.654763 4466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:17.658947 4466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:26:17.659019 4466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:17.659044 4466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:17.659061 4466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:26:17.659077 4466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:26:17.664610 4466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:17.665766 4466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:17.666849 4466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:17.675259 4471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:17.675259 4472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:17.675390 4466 server_base.cc:1034] running on GCE node
W20260929 16:26:17.675552 4474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:17.676069 4466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:17.677146 4466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:17.678303 4466 hybrid_clock.cc:648] HybridClock initialized: now 1790699177678273 us; error 45 us; skew 500 ppm
Sep 29 16:26:17 dist-test-slave-vq9v krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:17.681067 4466 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:26:17.681859 4466 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:17.681919 4466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:17.682128 4466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:17.684155 4466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "4cc82bf91f834d018b5951f81ebffdb6"
format_stamp: "Formatted at 2026-09-29 16:26:17 on dist-test-slave-vq9v"
server_key: "b1d49733aac97c8a4b2c23d2e64e082f"
server_key_iv: "0767e9ed811210f3beb4d3a9bb17a8a4"
server_key_version: "encryptionkey@0"
I20260929 16:26:17.684705 4466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "4cc82bf91f834d018b5951f81ebffdb6"
format_stamp: "Formatted at 2026-09-29 16:26:17 on dist-test-slave-vq9v"
server_key: "b1d49733aac97c8a4b2c23d2e64e082f"
server_key_iv: "0767e9ed811210f3beb4d3a9bb17a8a4"
server_key_version: "encryptionkey@0"
I20260929 16:26:17.688292 4466 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.005s
I20260929 16:26:17.690728 4480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:17.692071 4466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:26:17.692207 4466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "4cc82bf91f834d018b5951f81ebffdb6"
format_stamp: "Formatted at 2026-09-29 16:26:17 on dist-test-slave-vq9v"
server_key: "b1d49733aac97c8a4b2c23d2e64e082f"
server_key_iv: "0767e9ed811210f3beb4d3a9bb17a8a4"
server_key_version: "encryptionkey@0"
I20260929 16:26:17.692317 4466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:17.707094 4466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:17.710598 4466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:17.710825 4466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:17.718209 4466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43683
I20260929 16:26:17.718534 4532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43683 every 8 connection(s)
I20260929 16:26:17.718940 4466 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:26:17.721871 4533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:17.723744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 4466
I20260929 16:26:17.723869 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:26:17.724174 392 external_mini_cluster.cc:1428] Setting key 9bfebd1980e356a0610609f8cc642205
I20260929 16:26:17.728806 4533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6: Bootstrap starting.
I20260929 16:26:17.731196 4533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:17.732002 4533 log.cc:824] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:26:17 dist-test-slave-vq9v krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:17.734252 4533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6: No bootstrap required, opened a new log
I20260929 16:26:17.737859 4533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc82bf91f834d018b5951f81ebffdb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43683 } }
I20260929 16:26:17.738123 4533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:17.738179 4533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc82bf91f834d018b5951f81ebffdb6, State: Initialized, Role: FOLLOWER
I20260929 16:26:17.738616 4533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc82bf91f834d018b5951f81ebffdb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43683 } }
I20260929 16:26:17.738770 4533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:26:17.738852 4533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:26:17.738967 4533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:17.739351 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:17.725708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50584 (local address 127.0.98.62:43683)
0929 16:26:17.726219 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:17.726235 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:17.726275 (+ 40us) server_negotiation.cc:407] Connection header received
0929 16:26:17.726933 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:17.726950 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:17.727303 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:17.727654 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:17.728582 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:17.729840 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:17.730704 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:17.731200 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:17.731376 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:17.734219 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:17.734225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:17.734240 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:17.734268 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:17.736133 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:17.736613 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:17.736619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:17.736626 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:17.736763 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:17.737063 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:17.737067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:17.737069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:17.737541 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:17.737754 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:17.738041 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 16:26:17.738415 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":155,"threads_started":1}
I20260929 16:26:17.740180 4533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc82bf91f834d018b5951f81ebffdb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43683 } }
I20260929 16:26:17.740547 4533 leader_election.cc:302] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cc82bf91f834d018b5951f81ebffdb6; no voters:
I20260929 16:26:17.740887 4533 leader_election.cc:288] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:26:17.741003 4538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:17.741235 4538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6 [term 1 LEADER]: Becoming Leader. State: Replica: 4cc82bf91f834d018b5951f81ebffdb6, State: Running, Role: LEADER
I20260929 16:26:17.741570 4538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc82bf91f834d018b5951f81ebffdb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43683 } }
I20260929 16:26:17.742245 4533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:26:17.742873 4540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cc82bf91f834d018b5951f81ebffdb6. Latest consensus state: current_term: 1 leader_uuid: "4cc82bf91f834d018b5951f81ebffdb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc82bf91f834d018b5951f81ebffdb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43683 } } }
I20260929 16:26:17.743013 4540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:17.743330 4539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cc82bf91f834d018b5951f81ebffdb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc82bf91f834d018b5951f81ebffdb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43683 } } }
I20260929 16:26:17.743450 4539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:17.745584 4546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:26:17.748322 4546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:26:17.754249 4546 catalog_manager.cc:1347] Generated new cluster ID: 6fc338ccbd2a4d1982fbe7eed950394d
I20260929 16:26:17.754343 4546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:26:17.766109 4546 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:26:17.767432 4546 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:26:17.782565 4546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6: Generated new TSK 0
I20260929 16:26:17.783418 4546 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 16:26:17.817286 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43683
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:17.933503 4559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:17.933840 4559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:17.933890 4559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:17.938155 4559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:17.938237 4559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:17.938305 4559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:26:17.944196 4559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:17.945448 4559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:17.946489 4559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:26:17.957252 4559 server_base.cc:1034] running on GCE node
W20260929 16:26:17.957263 4564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:17.957270 4567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:17.957618 4565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:17.958184 4559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:17.958912 4559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:17.960155 4559 hybrid_clock.cc:648] HybridClock initialized: now 1790699177960133 us; error 39 us; skew 500 ppm
Sep 29 16:26:17 dist-test-slave-vq9v krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:17.963536 4559 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:26:17.964386 4559 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:17.964488 4559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:17.964735 4559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:17.966972 4559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "6ad01a444f7b48ed9b2e2da0a6ff41f6"
format_stamp: "Formatted at 2026-09-29 16:26:17 on dist-test-slave-vq9v"
server_key: "d0750b3ec42a66763a3f8a3445af204c"
server_key_iv: "a3e9c872ce1f6a105ea9332796125037"
server_key_version: "encryptionkey@0"
I20260929 16:26:17.967622 4559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6ad01a444f7b48ed9b2e2da0a6ff41f6"
format_stamp: "Formatted at 2026-09-29 16:26:17 on dist-test-slave-vq9v"
server_key: "d0750b3ec42a66763a3f8a3445af204c"
server_key_iv: "a3e9c872ce1f6a105ea9332796125037"
server_key_version: "encryptionkey@0"
I20260929 16:26:17.971776 4559 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:26:17.974514 4573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:17.975875 4559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:26:17.976117 4559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "6ad01a444f7b48ed9b2e2da0a6ff41f6"
format_stamp: "Formatted at 2026-09-29 16:26:17 on dist-test-slave-vq9v"
server_key: "d0750b3ec42a66763a3f8a3445af204c"
server_key_iv: "a3e9c872ce1f6a105ea9332796125037"
server_key_version: "encryptionkey@0"
I20260929 16:26:17.976246 4559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:17.991760 4559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:17.995376 4559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:17.995608 4559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:17.996279 4559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:17.997149 4559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:17.997251 4559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:17.997340 4559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:17.997396 4559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:18.006863 4559 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33255
I20260929 16:26:18.006881 4686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33255 every 8 connection(s)
I20260929 16:26:18.007507 4559 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:26:18.014487 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 4559
I20260929 16:26:18.014667 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:26:18.014958 392 external_mini_cluster.cc:1428] Setting key fa5f2114ee004c5c1015a01e6f850a66
Sep 29 16:26:18 dist-test-slave-vq9v krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:18.023011 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:18.009455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48309 (local address 127.0.98.62:43683)
0929 16:26:18.009614 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:18.009618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:18.010578 (+ 960us) server_negotiation.cc:407] Connection header received
0929 16:26:18.012083 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:18.012086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:18.012145 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:18.012237 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:18.013563 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.014114 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.015522 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.015812 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.015920 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:18.018948 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:18.018952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:18.018954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:18.018980 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:18.020583 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:18.021265 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:18.021269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:18.021271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:18.021361 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:18.021826 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:18.021830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:18.021833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:18.022043 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:18.022110 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:18.022683 (+ 573us) server_negotiation.cc:299] Negotiation successful
0929 16:26:18.022835 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:26:18.023862 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:18.009891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43683 (local address 127.0.98.1:48309)
0929 16:26:18.010401 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:18.010447 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:18.011843 (+ 1396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:18.012417 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:18.012425 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:18.012834 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:18.013410 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:18.013427 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.014277 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.014280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:18.015379 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:18.015395 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.015999 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.016002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:18.016241 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:18.017001 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:18.017019 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:18.018752 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:18.020721 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:18.020727 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:18.020747 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:18.021109 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:18.021478 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:18.021481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:18.021483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:18.021692 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:18.022168 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:18.022174 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:18.022430 (+ 256us) client_negotiation.cc:769] Sending connection context
0929 16:26:18.022629 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 16:26:18.022994 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 16:26:18.025178 4687 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43683
I20260929 16:26:18.025482 4687 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:18.025956 4687 heartbeater.cc:511] Master 127.0.98.62:43683 requested a full tablet report, sending...
I20260929 16:26:18.027536 4497 ts_manager.cc:194] Registered new tserver with Master: 6ad01a444f7b48ed9b2e2da0a6ff41f6 (127.0.98.1:33255)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 16:26:18.028795 4497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 16:26:18.045271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43683
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:18.159430 4692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:18.159703 4692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:18.159750 4692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:18.164422 4692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:18.164505 4692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:18.164611 4692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:26:18.170137 4692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:18.171360 4692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:18.172449 4692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:18.182428 4697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:18.182425 4700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:18.182425 4698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:18.182727 4692 server_base.cc:1034] running on GCE node
I20260929 16:26:18.183147 4692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:18.183835 4692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:18.185055 4692 hybrid_clock.cc:648] HybridClock initialized: now 1790699178185011 us; error 62 us; skew 500 ppm
Sep 29 16:26:18 dist-test-slave-vq9v krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:18.187955 4692 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:26:18.188658 4692 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:18.188723 4692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:18.188890 4692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:18.190925 4692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "f3273c854078449889dd2d6c134b1c26"
format_stamp: "Formatted at 2026-09-29 16:26:18 on dist-test-slave-vq9v"
server_key: "88ce56ed5bb23ea6c3e58cabc0cf3ae9"
server_key_iv: "fea54b698213945a642e94529017fbf4"
server_key_version: "encryptionkey@0"
I20260929 16:26:18.191439 4692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f3273c854078449889dd2d6c134b1c26"
format_stamp: "Formatted at 2026-09-29 16:26:18 on dist-test-slave-vq9v"
server_key: "88ce56ed5bb23ea6c3e58cabc0cf3ae9"
server_key_iv: "fea54b698213945a642e94529017fbf4"
server_key_version: "encryptionkey@0"
I20260929 16:26:18.195204 4692 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 16:26:18.197986 4706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:18.199540 4692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:18.199693 4692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "f3273c854078449889dd2d6c134b1c26"
format_stamp: "Formatted at 2026-09-29 16:26:18 on dist-test-slave-vq9v"
server_key: "88ce56ed5bb23ea6c3e58cabc0cf3ae9"
server_key_iv: "fea54b698213945a642e94529017fbf4"
server_key_version: "encryptionkey@0"
I20260929 16:26:18.199826 4692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:18.215561 4692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:18.219410 4692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:18.219635 4692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:18.220283 4692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:18.221146 4692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:18.221192 4692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:18.221310 4692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:18.221343 4692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:18.230114 4692 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37385
I20260929 16:26:18.230131 4819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37385 every 8 connection(s)
I20260929 16:26:18.230731 4692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:26:18.231930 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 4692
I20260929 16:26:18.232079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:26:18.232421 392 external_mini_cluster.cc:1428] Setting key a2e47cc77198148ce9cfa681eae510c3
Sep 29 16:26:18 dist-test-slave-vq9v krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 16:26:18.247319 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:18.232643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43717 (local address 127.0.98.62:43683)
0929 16:26:18.232815 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:18.232826 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:18.233825 (+ 999us) server_negotiation.cc:407] Connection header received
0929 16:26:18.235268 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:18.235275 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:18.235345 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:18.235460 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:18.237265 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.238279 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.239640 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.239871 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.239987 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:18.242691 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:18.242694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:18.242696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:18.242721 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:18.244593 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:18.245313 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:18.245320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:18.245325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:18.245433 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:18.245891 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:18.245898 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:18.245904 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:18.246190 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:18.246306 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:18.247014 (+ 708us) server_negotiation.cc:299] Negotiation successful
0929 16:26:18.247152 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:26:18.248246 4822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:18.233151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43683 (local address 127.0.98.2:43717)
0929 16:26:18.233674 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:18.233728 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:18.235043 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:18.235732 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:18.235744 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:18.236294 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:18.237018 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:18.237036 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.238458 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.238462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:18.239494 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:18.239504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.240030 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.240034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:18.240256 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:18.240891 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:18.240908 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:18.242399 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:18.244722 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:18.244728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:18.244745 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:18.245136 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:18.245560 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:18.245564 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:18.245565 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:18.245763 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:18.246316 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:18.246326 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:18.246697 (+ 371us) client_negotiation.cc:769] Sending connection context
0929 16:26:18.246936 (+ 239us) client_negotiation.cc:240] Negotiation successful
0929 16:26:18.247343 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":134,"threads_started":1}
I20260929 16:26:18.250231 4820 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43683
I20260929 16:26:18.250514 4820 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:18.251107 4820 heartbeater.cc:511] Master 127.0.98.62:43683 requested a full tablet report, sending...
I20260929 16:26:18.252584 4497 ts_manager.cc:194] Registered new tserver with Master: f3273c854078449889dd2d6c134b1c26 (127.0.98.2:37385)
I20260929 16:26:18.253589 4497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260929 16:26:18.601125 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.367s user 0.001s sys 0.002s
I20260929 16:26:18.601656 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43683
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:18.713583 4825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:18.713932 4825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:18.713997 4825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:18.718338 4825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:18.718434 4825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:18.718529 4825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:26:18.724167 4825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:18.725373 4825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:18.726403 4825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:18.736358 4830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:18.736359 4833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:18.736361 4831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:18.736583 4825 server_base.cc:1034] running on GCE node
I20260929 16:26:18.737128 4825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:18.737884 4825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:18.739079 4825 hybrid_clock.cc:648] HybridClock initialized: now 1790699178739064 us; error 31 us; skew 500 ppm
Sep 29 16:26:18 dist-test-slave-vq9v krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:18.753966 4825 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:26:18.754899 4825 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:18.754973 4825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:18.755210 4825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:18.757512 4825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "9565bfe9441e4077a92be84b9545b865"
format_stamp: "Formatted at 2026-09-29 16:26:18 on dist-test-slave-vq9v"
server_key: "e0e47a336c273fb47f1f7a6de3eafa77"
server_key_iv: "f4f987b8f6377af119623ce5b6cb3381"
server_key_version: "encryptionkey@0"
I20260929 16:26:18.758144 4825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9565bfe9441e4077a92be84b9545b865"
format_stamp: "Formatted at 2026-09-29 16:26:18 on dist-test-slave-vq9v"
server_key: "e0e47a336c273fb47f1f7a6de3eafa77"
server_key_iv: "f4f987b8f6377af119623ce5b6cb3381"
server_key_version: "encryptionkey@0"
I20260929 16:26:18.762079 4825 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:26:18.764786 4839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:18.766235 4825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:18.766382 4825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "9565bfe9441e4077a92be84b9545b865"
format_stamp: "Formatted at 2026-09-29 16:26:18 on dist-test-slave-vq9v"
server_key: "e0e47a336c273fb47f1f7a6de3eafa77"
server_key_iv: "f4f987b8f6377af119623ce5b6cb3381"
server_key_version: "encryptionkey@0"
I20260929 16:26:18.766503 4825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:18.794094 4825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:18.797902 4825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:18.798126 4825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:18.798751 4825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:18.799630 4825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:18.799700 4825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:18.799773 4825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:18.799817 4825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:18.809253 4825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40339
I20260929 16:26:18.809270 4952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40339 every 8 connection(s)
I20260929 16:26:18.809872 4825 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:26:18.818893 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 4825
I20260929 16:26:18.819063 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:26:18.819376 392 external_mini_cluster.cc:1428] Setting key cace5019460d159e55355047c9c0d05d
Sep 29 16:26:18 dist-test-slave-vq9v krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:18.824679 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:18.811658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53881 (local address 127.0.98.62:43683)
0929 16:26:18.811972 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:18.811977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:18.812732 (+ 755us) server_negotiation.cc:407] Connection header received
0929 16:26:18.814189 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:18.814191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:18.814240 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:18.814315 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:18.815631 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.816360 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.817568 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.817802 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.817896 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:18.820909 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:18.820913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:18.820916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:18.820949 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:18.822558 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:18.823105 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:18.823110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:18.823112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:18.823198 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:18.823542 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:18.823547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:18.823549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:18.823747 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:18.823855 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:18.824366 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 16:26:18.824493 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":110,"threads_started":1}
I20260929 16:26:18.825567 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:18.812108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43683 (local address 127.0.98.3:53881)
0929 16:26:18.812567 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:18.812615 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:18.813941 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:18.814481 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:18.814488 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:18.814881 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:18.815442 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:18.815456 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.816472 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.816475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:18.817416 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:18.817425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.817944 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.817947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:18.818129 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:18.818732 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:18.818755 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:18.820745 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:18.822708 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:18.822714 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:18.822731 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:18.822997 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:18.823292 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:18.823296 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:18.823298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:18.823445 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:18.823851 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:18.823859 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:18.824132 (+ 273us) client_negotiation.cc:769] Sending connection context
0929 16:26:18.824329 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 16:26:18.824662 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":101,"threads_started":1}
I20260929 16:26:18.826620 4953 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43683
I20260929 16:26:18.826891 4953 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:18.827364 4953 heartbeater.cc:511] Master 127.0.98.62:43683 requested a full tablet report, sending...
I20260929 16:26:18.828490 4497 ts_manager.cc:194] Registered new tserver with Master: 9565bfe9441e4077a92be84b9545b865 (127.0.98.3:40339)
I20260929 16:26:18.829130 4497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53881
I20260929 16:26:18.833855 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:26:18.844278 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:18.836040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50590 (local address 127.0.98.62:43683)
0929 16:26:18.836223 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:18.836228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:18.836352 (+ 124us) server_negotiation.cc:407] Connection header received
0929 16:26:18.836525 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:18.836528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:18.836571 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:18.836658 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:18.837460 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.837991 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.838855 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.839108 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.839226 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:18.840642 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:18.840646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:18.840649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:18.840683 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:18.842302 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:18.842757 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:18.842761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:18.842763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:18.842859 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:18.843224 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:18.843228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:18.843231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:18.843489 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:18.843628 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:18.843946 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 16:26:18.844120 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 16:26:18.847570 4497 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:26:18.850683 4497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:26:18.866159 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:18.860232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37385 (local address 127.0.0.1:43162)
0929 16:26:18.860961 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:18.860974 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:18.861062 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:18.861545 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:18.861548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:18.861567 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:18.861654 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:18.861659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.862933 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.862937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:18.864498 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:18.864510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.865548 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.865552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:18.865668 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:18.865725 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 16:26:18.865842 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 16:26:18.865978 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":23,"thread_start_us":84,"threads_started":1}
I20260929 16:26:18.866212 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:18.859403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40339 (local address 127.0.0.1:35094)
0929 16:26:18.859844 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:18.859890 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:18.860025 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:18.860916 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:18.860923 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:18.860945 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:18.861108 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:18.861118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.862405 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.862408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:18.864686 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:18.864694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.865787 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.865791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:18.865899 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:18.865933 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:26:18.865992 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 16:26:18.866064 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":6,"thread_start_us":160,"threads_started":1}
I20260929 16:26:18.866778 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:18.860522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33255 (local address 127.0.0.1:37680)
0929 16:26:18.861260 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:18.861273 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:18.861346 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:18.861967 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:18.861970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:18.861988 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:18.862090 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:18.862097 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.863523 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.863526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:18.864914 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:18.864922 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.866388 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.866392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:18.866478 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:18.866499 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:26:18.866562 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 16:26:18.866646 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":27,"thread_start_us":78,"threads_started":1}
I20260929 16:26:18.866971 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:18.859567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35094 (local address 127.0.98.3:40339)
0929 16:26:18.859867 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:18.859880 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:18.859964 (+ 84us) server_negotiation.cc:407] Connection header received
0929 16:26:18.860317 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:18.860327 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:18.860547 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:18.860710 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:18.861257 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.862257 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.864812 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.865668 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.865794 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:18.865969 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:18.866037 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 16:26:18.866132 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":120,"threads_started":1}
I20260929 16:26:18.867069 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:18.860373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43162 (local address 127.0.98.2:37385)
0929 16:26:18.860862 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:18.860869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:18.861075 (+ 206us) server_negotiation.cc:407] Connection header received
0929 16:26:18.861162 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:18.861167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:18.861392 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:18.861559 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:18.861754 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.862785 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.864632 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.865406 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.865524 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:18.865676 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:18.866656 (+ 980us) server_negotiation.cc:299] Negotiation successful
0929 16:26:18.866768 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":110,"threads_started":1}
I20260929 16:26:18.867244 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:18.860946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37680 (local address 127.0.98.1:33255)
0929 16:26:18.861368 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:18.861375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:18.861399 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:26:18.861474 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:18.861482 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:18.861679 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:18.861831 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:18.862225 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.863384 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.865054 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:18.865848 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:18.865952 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:18.866899 (+ 947us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:18.866972 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 16:26:18.867079 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":104,"threads_started":1}
I20260929 16:26:18.869890 4887 tablet_service.cc:1467] Processing CreateTablet for tablet ea64f6d48619470b80ada9fe92fab8df (DEFAULT_TABLE table=test-table [id=8d531e7bc5244323988091352fc93b15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:18.869890 4754 tablet_service.cc:1467] Processing CreateTablet for tablet ea64f6d48619470b80ada9fe92fab8df (DEFAULT_TABLE table=test-table [id=8d531e7bc5244323988091352fc93b15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:18.869837 4621 tablet_service.cc:1467] Processing CreateTablet for tablet ea64f6d48619470b80ada9fe92fab8df (DEFAULT_TABLE table=test-table [id=8d531e7bc5244323988091352fc93b15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:18.871011 4887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea64f6d48619470b80ada9fe92fab8df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:18.871004 4754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea64f6d48619470b80ada9fe92fab8df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:18.871017 4621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea64f6d48619470b80ada9fe92fab8df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:18.877632 4972 tablet_bootstrap.cc:492] T ea64f6d48619470b80ada9fe92fab8df P 6ad01a444f7b48ed9b2e2da0a6ff41f6: Bootstrap starting.
I20260929 16:26:18.878649 4973 tablet_bootstrap.cc:492] T ea64f6d48619470b80ada9fe92fab8df P f3273c854078449889dd2d6c134b1c26: Bootstrap starting.
I20260929 16:26:18.880115 4972 tablet_bootstrap.cc:654] T ea64f6d48619470b80ada9fe92fab8df P 6ad01a444f7b48ed9b2e2da0a6ff41f6: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:18.880596 4973 tablet_bootstrap.cc:654] T ea64f6d48619470b80ada9fe92fab8df P f3273c854078449889dd2d6c134b1c26: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:18.881151 4972 log.cc:824] T ea64f6d48619470b80ada9fe92fab8df P 6ad01a444f7b48ed9b2e2da0a6ff41f6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:18.881425 4973 log.cc:824] T ea64f6d48619470b80ada9fe92fab8df P f3273c854078449889dd2d6c134b1c26: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:18.882144 4974 tablet_bootstrap.cc:492] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865: Bootstrap starting.
I20260929 16:26:18.883379 4972 tablet_bootstrap.cc:492] T ea64f6d48619470b80ada9fe92fab8df P 6ad01a444f7b48ed9b2e2da0a6ff41f6: No bootstrap required, opened a new log
I20260929 16:26:18.883531 4973 tablet_bootstrap.cc:492] T ea64f6d48619470b80ada9fe92fab8df P f3273c854078449889dd2d6c134b1c26: No bootstrap required, opened a new log
I20260929 16:26:18.883638 4972 ts_tablet_manager.cc:1397] T ea64f6d48619470b80ada9fe92fab8df P 6ad01a444f7b48ed9b2e2da0a6ff41f6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:26:18.883709 4973 ts_tablet_manager.cc:1397] T ea64f6d48619470b80ada9fe92fab8df P f3273c854078449889dd2d6c134b1c26: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 16:26:18.884398 4974 tablet_bootstrap.cc:654] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:18.885783 4974 log.cc:824] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:18.888151 4974 tablet_bootstrap.cc:492] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865: No bootstrap required, opened a new log
I20260929 16:26:18.887954 4972 raft_consensus.cc:348] T ea64f6d48619470b80ada9fe92fab8df P 6ad01a444f7b48ed9b2e2da0a6ff41f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9565bfe9441e4077a92be84b9545b865" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40339 } } peers { permanent_uuid: "f3273c854078449889dd2d6c134b1c26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } } peers { permanent_uuid: "6ad01a444f7b48ed9b2e2da0a6ff41f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33255 } }
I20260929 16:26:18.888073 4973 raft_consensus.cc:348] T ea64f6d48619470b80ada9fe92fab8df P f3273c854078449889dd2d6c134b1c26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9565bfe9441e4077a92be84b9545b865" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40339 } } peers { permanent_uuid: "f3273c854078449889dd2d6c134b1c26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } } peers { permanent_uuid: "6ad01a444f7b48ed9b2e2da0a6ff41f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33255 } }
I20260929 16:26:18.888279 4972 raft_consensus.cc:374] T ea64f6d48619470b80ada9fe92fab8df P 6ad01a444f7b48ed9b2e2da0a6ff41f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:18.888339 4972 raft_consensus.cc:729] T ea64f6d48619470b80ada9fe92fab8df P 6ad01a444f7b48ed9b2e2da0a6ff41f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad01a444f7b48ed9b2e2da0a6ff41f6, State: Initialized, Role: FOLLOWER
I20260929 16:26:18.888343 4973 raft_consensus.cc:374] T ea64f6d48619470b80ada9fe92fab8df P f3273c854078449889dd2d6c134b1c26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:18.888350 4974 ts_tablet_manager.cc:1397] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:26:18.888401 4973 raft_consensus.cc:729] T ea64f6d48619470b80ada9fe92fab8df P f3273c854078449889dd2d6c134b1c26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3273c854078449889dd2d6c134b1c26, State: Initialized, Role: FOLLOWER
I20260929 16:26:18.888837 4972 consensus_queue.cc:260] T ea64f6d48619470b80ada9fe92fab8df P 6ad01a444f7b48ed9b2e2da0a6ff41f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9565bfe9441e4077a92be84b9545b865" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40339 } } peers { permanent_uuid: "f3273c854078449889dd2d6c134b1c26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } } peers { permanent_uuid: "6ad01a444f7b48ed9b2e2da0a6ff41f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33255 } }
I20260929 16:26:18.888860 4973 consensus_queue.cc:260] T ea64f6d48619470b80ada9fe92fab8df P f3273c854078449889dd2d6c134b1c26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9565bfe9441e4077a92be84b9545b865" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40339 } } peers { permanent_uuid: "f3273c854078449889dd2d6c134b1c26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } } peers { permanent_uuid: "6ad01a444f7b48ed9b2e2da0a6ff41f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33255 } }
I20260929 16:26:18.889847 4687 heartbeater.cc:503] Master 127.0.98.62:43683 was elected leader, sending a full tablet report...
I20260929 16:26:18.889936 4973 ts_tablet_manager.cc:1428] T ea64f6d48619470b80ada9fe92fab8df P f3273c854078449889dd2d6c134b1c26: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:26:18.890275 4820 heartbeater.cc:503] Master 127.0.98.62:43683 was elected leader, sending a full tablet report...
I20260929 16:26:18.890556 4972 ts_tablet_manager.cc:1428] T ea64f6d48619470b80ada9fe92fab8df P 6ad01a444f7b48ed9b2e2da0a6ff41f6: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 16:26:18.893450 4974 raft_consensus.cc:348] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9565bfe9441e4077a92be84b9545b865" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40339 } } peers { permanent_uuid: "f3273c854078449889dd2d6c134b1c26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } } peers { permanent_uuid: "6ad01a444f7b48ed9b2e2da0a6ff41f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33255 } }
I20260929 16:26:18.893785 4974 raft_consensus.cc:374] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:18.893880 4974 raft_consensus.cc:729] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9565bfe9441e4077a92be84b9545b865, State: Initialized, Role: FOLLOWER
I20260929 16:26:18.894394 4974 consensus_queue.cc:260] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9565bfe9441e4077a92be84b9545b865" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40339 } } peers { permanent_uuid: "f3273c854078449889dd2d6c134b1c26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } } peers { permanent_uuid: "6ad01a444f7b48ed9b2e2da0a6ff41f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33255 } }
I20260929 16:26:18.895177 4953 heartbeater.cc:503] Master 127.0.98.62:43683 was elected leader, sending a full tablet report...
I20260929 16:26:18.895488 4974 ts_tablet_manager.cc:1428] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260929 16:26:18.981952 4821 tablet.cc:2369] T ea64f6d48619470b80ada9fe92fab8df P f3273c854078449889dd2d6c134b1c26: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:26:19.009073 4688 tablet.cc:2369] T ea64f6d48619470b80ada9fe92fab8df P 6ad01a444f7b48ed9b2e2da0a6ff41f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:26:19.048168 4980 raft_consensus.cc:482] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:26:19.048385 4980 raft_consensus.cc:504] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9565bfe9441e4077a92be84b9545b865" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40339 } } peers { permanent_uuid: "f3273c854078449889dd2d6c134b1c26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } } peers { permanent_uuid: "6ad01a444f7b48ed9b2e2da0a6ff41f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33255 } }
I20260929 16:26:19.049553 4980 leader_election.cc:288] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3273c854078449889dd2d6c134b1c26 (127.0.98.2:37385), 6ad01a444f7b48ed9b2e2da0a6ff41f6 (127.0.98.1:33255)
I20260929 16:26:19.054019 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:19.049939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37385 (local address 127.0.98.3:41335)
0929 16:26:19.050105 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:19.050122 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:19.050222 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:19.050623 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:19.050626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:19.050643 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:19.050784 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:19.050790 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:19.051765 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:19.051769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:19.052636 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:19.052645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:19.053535 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:19.053538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:19.053669 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:19.053696 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:26:19.053782 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 16:26:19.053870 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":1}
I20260929 16:26:19.054042 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:19.050006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41335 (local address 127.0.98.2:37385)
0929 16:26:19.050163 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:19.050169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:19.050184 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:26:19.050335 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:19.050338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:19.050389 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:19.050461 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:19.050935 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:19.051633 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:19.052770 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:19.053418 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:19.053504 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:19.053569 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:19.053806 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 16:26:19.053905 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:26:19.054400 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:19.050430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56163 (local address 127.0.98.1:33255)
0929 16:26:19.050579 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:19.050584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:19.050717 (+ 133us) server_negotiation.cc:407] Connection header received
0929 16:26:19.050805 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:19.050809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:19.050865 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:19.050935 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:19.051278 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:19.051835 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:19.053258 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:19.053772 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:19.053866 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:19.053919 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:19.054176 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 16:26:19.054273 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:26:19.054745 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:19.050245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33255 (local address 127.0.98.3:56163)
0929 16:26:19.050621 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:19.050634 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:19.050725 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:19.050968 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:19.050971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:19.050988 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:19.051163 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:19.051170 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:19.051949 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:19.051952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:19.053072 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:19.053082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:19.053892 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:19.053896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:19.053997 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:19.054030 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:26:19.054553 (+ 523us) client_negotiation.cc:240] Negotiation successful
0929 16:26:19.054639 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":25,"thread_start_us":164,"threads_started":1}
I20260929 16:26:19.054805 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea64f6d48619470b80ada9fe92fab8df" candidate_uuid: "9565bfe9441e4077a92be84b9545b865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3273c854078449889dd2d6c134b1c26" is_pre_election: true
I20260929 16:26:19.055166 4774 raft_consensus.cc:2454] T ea64f6d48619470b80ada9fe92fab8df P f3273c854078449889dd2d6c134b1c26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9565bfe9441e4077a92be84b9545b865 in term 0.
I20260929 16:26:19.055500 4641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea64f6d48619470b80ada9fe92fab8df" candidate_uuid: "9565bfe9441e4077a92be84b9545b865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad01a444f7b48ed9b2e2da0a6ff41f6" is_pre_election: true
I20260929 16:26:19.055712 4641 raft_consensus.cc:2454] T ea64f6d48619470b80ada9fe92fab8df P 6ad01a444f7b48ed9b2e2da0a6ff41f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9565bfe9441e4077a92be84b9545b865 in term 0.
I20260929 16:26:19.055732 4841 leader_election.cc:302] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9565bfe9441e4077a92be84b9545b865, f3273c854078449889dd2d6c134b1c26; no voters:
I20260929 16:26:19.055985 4980 raft_consensus.cc:2781] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:26:19.056066 4980 raft_consensus.cc:482] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:26:19.056109 4980 raft_consensus.cc:3037] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:19.057184 4980 raft_consensus.cc:504] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9565bfe9441e4077a92be84b9545b865" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40339 } } peers { permanent_uuid: "f3273c854078449889dd2d6c134b1c26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } } peers { permanent_uuid: "6ad01a444f7b48ed9b2e2da0a6ff41f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33255 } }
I20260929 16:26:19.057590 4980 leader_election.cc:288] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865 [CANDIDATE]: Term 1 election: Requested vote from peers f3273c854078449889dd2d6c134b1c26 (127.0.98.2:37385), 6ad01a444f7b48ed9b2e2da0a6ff41f6 (127.0.98.1:33255)
I20260929 16:26:19.058061 4641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea64f6d48619470b80ada9fe92fab8df" candidate_uuid: "9565bfe9441e4077a92be84b9545b865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad01a444f7b48ed9b2e2da0a6ff41f6"
I20260929 16:26:19.058060 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea64f6d48619470b80ada9fe92fab8df" candidate_uuid: "9565bfe9441e4077a92be84b9545b865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3273c854078449889dd2d6c134b1c26"
I20260929 16:26:19.058179 4641 raft_consensus.cc:3037] T ea64f6d48619470b80ada9fe92fab8df P 6ad01a444f7b48ed9b2e2da0a6ff41f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:19.058216 4774 raft_consensus.cc:3037] T ea64f6d48619470b80ada9fe92fab8df P f3273c854078449889dd2d6c134b1c26 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:19.059229 4641 raft_consensus.cc:2454] T ea64f6d48619470b80ada9fe92fab8df P 6ad01a444f7b48ed9b2e2da0a6ff41f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9565bfe9441e4077a92be84b9545b865 in term 1.
I20260929 16:26:19.059382 4774 raft_consensus.cc:2454] T ea64f6d48619470b80ada9fe92fab8df P f3273c854078449889dd2d6c134b1c26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9565bfe9441e4077a92be84b9545b865 in term 1.
I20260929 16:26:19.059597 4841 leader_election.cc:302] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad01a444f7b48ed9b2e2da0a6ff41f6, 9565bfe9441e4077a92be84b9545b865; no voters:
I20260929 16:26:19.059808 4980 raft_consensus.cc:2781] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:19.060032 4980 raft_consensus.cc:686] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865 [term 1 LEADER]: Becoming Leader. State: Replica: 9565bfe9441e4077a92be84b9545b865, State: Running, Role: LEADER
I20260929 16:26:19.060395 4980 consensus_queue.cc:237] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9565bfe9441e4077a92be84b9545b865" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40339 } } peers { permanent_uuid: "f3273c854078449889dd2d6c134b1c26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } } peers { permanent_uuid: "6ad01a444f7b48ed9b2e2da0a6ff41f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33255 } }
W20260929 16:26:19.060751 4954 tablet.cc:2369] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:26:19.063868 4496 catalog_manager.cc:5495] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9565bfe9441e4077a92be84b9545b865 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9565bfe9441e4077a92be84b9545b865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9565bfe9441e4077a92be84b9545b865" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3273c854078449889dd2d6c134b1c26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ad01a444f7b48ed9b2e2da0a6ff41f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33255 } health_report { overall_health: UNKNOWN } } }
I20260929 16:26:19.102679 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:19.099027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.0.98.3:40339)
0929 16:26:19.099199 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:19.099205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:19.099224 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:26:19.099313 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:19.099316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:19.099370 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:19.099487 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:19.099760 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:19.100293 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:19.101159 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:19.101442 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:19.101551 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:19.101868 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:19.102284 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:19.102421 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:19.102458 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 16:26:19.102513 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:26:19.108861 4641 raft_consensus.cc:1261] T ea64f6d48619470b80ada9fe92fab8df P 6ad01a444f7b48ed9b2e2da0a6ff41f6 [term 1 FOLLOWER]: Refusing update from remote peer 9565bfe9441e4077a92be84b9545b865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:19.108861 4774 raft_consensus.cc:1261] T ea64f6d48619470b80ada9fe92fab8df P f3273c854078449889dd2d6c134b1c26 [term 1 FOLLOWER]: Refusing update from remote peer 9565bfe9441e4077a92be84b9545b865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:19.109661 4982 consensus_queue.cc:1035] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3273c854078449889dd2d6c134b1c26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:19.109838 4980 consensus_queue.cc:1035] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad01a444f7b48ed9b2e2da0a6ff41f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:19.127290 4497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:19.127458 4497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:19.130311 4497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4cc82bf91f834d018b5951f81ebffdb6: Sending DeleteTablet for 3 replicas of tablet ea64f6d48619470b80ada9fe92fab8df
I20260929 16:26:19.131170 4887 tablet_service.cc:1514] Processing DeleteTablet for tablet ea64f6d48619470b80ada9fe92fab8df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35094
I20260929 16:26:19.131378 4621 tablet_service.cc:1514] Processing DeleteTablet for tablet ea64f6d48619470b80ada9fe92fab8df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37680
I20260929 16:26:19.131529 4754 tablet_service.cc:1514] Processing DeleteTablet for tablet ea64f6d48619470b80ada9fe92fab8df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43162
I20260929 16:26:19.131904 4995 tablet_replica.cc:330] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865: stopping tablet replica
I20260929 16:26:19.131947 4997 tablet_replica.cc:330] T ea64f6d48619470b80ada9fe92fab8df P f3273c854078449889dd2d6c134b1c26: stopping tablet replica
I20260929 16:26:19.132084 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 4559
I20260929 16:26:19.132252 4995 raft_consensus.cc:2229] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:26:19.132655 4995 raft_consensus.cc:2258] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:19.134064 4997 raft_consensus.cc:2229] T ea64f6d48619470b80ada9fe92fab8df P f3273c854078449889dd2d6c134b1c26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:19.134423 4997 raft_consensus.cc:2258] T ea64f6d48619470b80ada9fe92fab8df P f3273c854078449889dd2d6c134b1c26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:19.135658 4995 ts_tablet_manager.cc:1905] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:19.135687 4997 ts_tablet_manager.cc:1905] T ea64f6d48619470b80ada9fe92fab8df P f3273c854078449889dd2d6c134b1c26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:19.138434 4997 ts_tablet_manager.cc:1918] T ea64f6d48619470b80ada9fe92fab8df P f3273c854078449889dd2d6c134b1c26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:26:19.138434 4995 ts_tablet_manager.cc:1918] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:26:19.138543 4995 log.cc:1196] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/ea64f6d48619470b80ada9fe92fab8df
I20260929 16:26:19.138545 4997 log.cc:1196] T ea64f6d48619470b80ada9fe92fab8df P f3273c854078449889dd2d6c134b1c26: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/ea64f6d48619470b80ada9fe92fab8df
I20260929 16:26:19.138876 4997 ts_tablet_manager.cc:1939] T ea64f6d48619470b80ada9fe92fab8df P f3273c854078449889dd2d6c134b1c26: Deleting consensus metadata
I20260929 16:26:19.138876 4995 ts_tablet_manager.cc:1939] T ea64f6d48619470b80ada9fe92fab8df P 9565bfe9441e4077a92be84b9545b865: Deleting consensus metadata
I20260929 16:26:19.140172 4482 catalog_manager.cc:4841] TS 9565bfe9441e4077a92be84b9545b865 (127.0.98.3:40339): tablet ea64f6d48619470b80ada9fe92fab8df (table test-table [id=8d531e7bc5244323988091352fc93b15]) successfully deleted
I20260929 16:26:19.140417 4482 catalog_manager.cc:4841] TS f3273c854078449889dd2d6c134b1c26 (127.0.98.2:37385): tablet ea64f6d48619470b80ada9fe92fab8df (table test-table [id=8d531e7bc5244323988091352fc93b15]) successfully deleted
W20260929 16:26:19.140875 4482 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33255 (error 108)
I20260929 16:26:19.141045 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 4692
I20260929 16:26:19.143528 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:19.143211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33255 (local address 127.0.0.1:37696)
0929 16:26:19.143353 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:19.143433 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 16:26:19.143684 4482 catalog_manager.cc:4584] TS 6ad01a444f7b48ed9b2e2da0a6ff41f6 (127.0.98.1:33255): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea64f6d48619470b80ada9fe92fab8df: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33255: connect: Connection refused (error 111)
I20260929 16:26:19.147095 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 4825
I20260929 16:26:19.152141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 4466
2026-09-29T16:26:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2795 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T16:26:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:26:19Z Disabled control of system clock
I20260929 16:26:19.182606 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32885
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 16:26:19.295414 5002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:19.295691 5002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:19.295727 5002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:19.299896 5002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:26:19.299961 5002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:19.299978 5002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:19.299996 5002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:26:19.300011 5002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:26:19.304845 5002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32885
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:19.305989 5002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:19.307417 5002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:19.316771 5008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:19.316855 5010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:19.316825 5007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:19.317075 5002 server_base.cc:1034] running on GCE node
I20260929 16:26:19.317790 5002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:19.319031 5002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:19.320258 5002 hybrid_clock.cc:648] HybridClock initialized: now 1790699179320226 us; error 59 us; skew 500 ppm
I20260929 16:26:19.322522 5002 webserver.cc:460] Webserver started at http://127.0.98.62:37713/ using document root <none> and password file <none>
I20260929 16:26:19.323184 5002 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:19.323288 5002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:19.323555 5002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:19.325984 5002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "76ae60cb45af468abf8f04fa21d0e83d"
format_stamp: "Formatted at 2026-09-29 16:26:19 on dist-test-slave-vq9v"
server_key: "56bd8e0177814800cbe71901bc4b5d1e"
server_key_iv: "f0eab6ee6d2047620958834f39821c02"
server_key_version: "encryptionkey@0"
I20260929 16:26:19.326627 5002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "76ae60cb45af468abf8f04fa21d0e83d"
format_stamp: "Formatted at 2026-09-29 16:26:19 on dist-test-slave-vq9v"
server_key: "56bd8e0177814800cbe71901bc4b5d1e"
server_key_iv: "f0eab6ee6d2047620958834f39821c02"
server_key_version: "encryptionkey@0"
I20260929 16:26:19.330802 5002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:26:19.333853 5016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:19.335175 5002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:19.335335 5002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "76ae60cb45af468abf8f04fa21d0e83d"
format_stamp: "Formatted at 2026-09-29 16:26:19 on dist-test-slave-vq9v"
server_key: "56bd8e0177814800cbe71901bc4b5d1e"
server_key_iv: "f0eab6ee6d2047620958834f39821c02"
server_key_version: "encryptionkey@0"
I20260929 16:26:19.335467 5002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:19.340291 5002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:19.340950 5002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:19.341159 5002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:19.348397 5002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32885
I20260929 16:26:19.348412 5068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32885 every 8 connection(s)
I20260929 16:26:19.349488 5002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:26:19.353114 5069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:19.358750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 5002
I20260929 16:26:19.358896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:26:19.359179 392 external_mini_cluster.cc:1428] Setting key 7c97a42b5dab622ae1cd332b96617734
I20260929 16:26:19.359774 5069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d: Bootstrap starting.
I20260929 16:26:19.362493 5069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:19.363302 5069 log.cc:824] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:19.365365 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:19.360792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34108 (local address 127.0.98.62:32885)
0929 16:26:19.361601 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:19.361620 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:19.361658 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:26:19.362377 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:19.362395 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:19.362825 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:19.363179 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:19.363973 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:19.363979 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:19.363993 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:19.364011 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:19.364064 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:19.364080 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:19.364197 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:19.364331 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 16:26:19.364579 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":160,"threads_started":1}
I20260929 16:26:19.365639 5069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d: No bootstrap required, opened a new log
I20260929 16:26:19.370486 5069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ae60cb45af468abf8f04fa21d0e83d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32885 } }
I20260929 16:26:19.370731 5069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:19.370793 5069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76ae60cb45af468abf8f04fa21d0e83d, State: Initialized, Role: FOLLOWER
I20260929 16:26:19.371285 5069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ae60cb45af468abf8f04fa21d0e83d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32885 } }
I20260929 16:26:19.371419 5069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:26:19.371510 5069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:26:19.371618 5069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:19.372664 5069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ae60cb45af468abf8f04fa21d0e83d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32885 } }
I20260929 16:26:19.373044 5069 leader_election.cc:302] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76ae60cb45af468abf8f04fa21d0e83d; no voters:
I20260929 16:26:19.373414 5069 leader_election.cc:288] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:26:19.373551 5074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:19.373826 5074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d [term 1 LEADER]: Becoming Leader. State: Replica: 76ae60cb45af468abf8f04fa21d0e83d, State: Running, Role: LEADER
I20260929 16:26:19.374166 5074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ae60cb45af468abf8f04fa21d0e83d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32885 } }
I20260929 16:26:19.374823 5069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:26:19.375348 5075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76ae60cb45af468abf8f04fa21d0e83d. Latest consensus state: current_term: 1 leader_uuid: "76ae60cb45af468abf8f04fa21d0e83d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ae60cb45af468abf8f04fa21d0e83d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32885 } } }
I20260929 16:26:19.375494 5075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:19.375815 5076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76ae60cb45af468abf8f04fa21d0e83d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76ae60cb45af468abf8f04fa21d0e83d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32885 } } }
I20260929 16:26:19.375928 5076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:19.376245 5078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:26:19.379706 5078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:26:19.386866 5078 catalog_manager.cc:1347] Generated new cluster ID: 22c4121c0db7400c81489dc3962cc1d5
I20260929 16:26:19.386973 5078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:26:19.395087 5078 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:26:19.395467 5078 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:26:19.403235 5078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d: Generated new TSK 0
I20260929 16:26:19.404024 5078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:26:19.412662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32885
--builtin_ntp_servers=127.0.98.20:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 16:26:19.533162 5093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:19.533474 5093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:19.533515 5093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:19.538587 5093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:19.538661 5093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:19.538729 5093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:26:19.544422 5093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:19.545671 5093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:19.547113 5093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:19.557830 5098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:19.557821 5099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:19.557988 5093 server_base.cc:1034] running on GCE node
W20260929 16:26:19.557821 5101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:19.558581 5093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:19.559270 5093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:19.560503 5093 hybrid_clock.cc:648] HybridClock initialized: now 1790699179560471 us; error 44 us; skew 500 ppm
I20260929 16:26:19.562772 5093 webserver.cc:460] Webserver started at http://127.0.98.1:33397/ using document root <none> and password file <none>
I20260929 16:26:19.563418 5093 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:19.563525 5093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:19.563788 5093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:19.566195 5093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "638c6b6277024213a854b9aff6f16c52"
format_stamp: "Formatted at 2026-09-29 16:26:19 on dist-test-slave-vq9v"
server_key: "8d870bb939b0a8eb6a9f7ee4b836256c"
server_key_iv: "a2e041b1d56622009119e81fc1579019"
server_key_version: "encryptionkey@0"
I20260929 16:26:19.566860 5093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "638c6b6277024213a854b9aff6f16c52"
format_stamp: "Formatted at 2026-09-29 16:26:19 on dist-test-slave-vq9v"
server_key: "8d870bb939b0a8eb6a9f7ee4b836256c"
server_key_iv: "a2e041b1d56622009119e81fc1579019"
server_key_version: "encryptionkey@0"
I20260929 16:26:19.571142 5093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:26:19.574554 5107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:19.576436 5093 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:26:19.576630 5093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "638c6b6277024213a854b9aff6f16c52"
format_stamp: "Formatted at 2026-09-29 16:26:19 on dist-test-slave-vq9v"
server_key: "8d870bb939b0a8eb6a9f7ee4b836256c"
server_key_iv: "a2e041b1d56622009119e81fc1579019"
server_key_version: "encryptionkey@0"
I20260929 16:26:19.576818 5093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:19.582201 5093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:19.582864 5093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:19.583040 5093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:19.583601 5093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:19.584564 5093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:19.584619 5093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:19.584664 5093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:19.584683 5093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:19.594969 5093 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38891
I20260929 16:26:19.594991 5220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38891 every 8 connection(s)
I20260929 16:26:19.596019 5093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:26:19.600286 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 5093
I20260929 16:26:19.600456 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:26:19.600759 392 external_mini_cluster.cc:1428] Setting key a7ad2193139a82c140b554ce921c0f46
I20260929 16:26:19.602784 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:19.597940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40801 (local address 127.0.98.62:32885)
0929 16:26:19.598074 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:19.598078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:19.599158 (+ 1080us) server_negotiation.cc:407] Connection header received
0929 16:26:19.600839 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:19.600847 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:19.600906 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:19.600991 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:19.602305 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:19.602309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:19.602312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:19.602325 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:19.602355 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:19.602364 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:19.602460 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:19.602607 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 16:26:19.602655 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:26:19.603153 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32885
--builtin_ntp_servers=127.0.98.20:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 16:26:19.603802 5223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:19.598320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32885 (local address 127.0.98.1:40801)
0929 16:26:19.598988 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:19.599030 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:19.600523 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:19.601284 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:19.601293 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:19.601398 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:19.601987 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:26:19.601989 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:19.602045 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:26:19.602139 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:19.602447 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:19.602456 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:19.602462 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:26:19.602666 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 16:26:19.602933 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":122,"threads_started":1}
I20260929 16:26:19.605751 5221 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32885
I20260929 16:26:19.606055 5221 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:19.606612 5221 heartbeater.cc:511] Master 127.0.98.62:32885 requested a full tablet report, sending...
I20260929 16:26:19.608878 5033 ts_manager.cc:194] Registered new tserver with Master: 638c6b6277024213a854b9aff6f16c52 (127.0.98.1:38891)
W20260929 16:26:19.722093 5224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:19.722388 5224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:19.722429 5224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:19.726683 5224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:19.726763 5224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:19.726824 5224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:26:19.732004 5224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:19.733115 5224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:19.734687 5224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:19.745539 5232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:19.745529 5230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:19.745529 5229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:19.746011 5224 server_base.cc:1034] running on GCE node
I20260929 16:26:19.746412 5224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:19.747108 5224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:19.748308 5224 hybrid_clock.cc:648] HybridClock initialized: now 1790699179748290 us; error 41 us; skew 500 ppm
I20260929 16:26:19.750582 5224 webserver.cc:460] Webserver started at http://127.0.98.2:42713/ using document root <none> and password file <none>
I20260929 16:26:19.751165 5224 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:19.751241 5224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:19.751426 5224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:19.753718 5224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "285f2041ddf64973a8dee7ca02020305"
format_stamp: "Formatted at 2026-09-29 16:26:19 on dist-test-slave-vq9v"
server_key: "f082212e1e4f7141edec571246cd496c"
server_key_iv: "dd33af80636aca5b00d0b8c84c044569"
server_key_version: "encryptionkey@0"
I20260929 16:26:19.754292 5224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "285f2041ddf64973a8dee7ca02020305"
format_stamp: "Formatted at 2026-09-29 16:26:19 on dist-test-slave-vq9v"
server_key: "f082212e1e4f7141edec571246cd496c"
server_key_iv: "dd33af80636aca5b00d0b8c84c044569"
server_key_version: "encryptionkey@0"
I20260929 16:26:19.758375 5224 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:26:19.761155 5238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:19.762854 5224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:26:19.763049 5224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "285f2041ddf64973a8dee7ca02020305"
format_stamp: "Formatted at 2026-09-29 16:26:19 on dist-test-slave-vq9v"
server_key: "f082212e1e4f7141edec571246cd496c"
server_key_iv: "dd33af80636aca5b00d0b8c84c044569"
server_key_version: "encryptionkey@0"
I20260929 16:26:19.763195 5224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:19.768193 5224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:19.768849 5224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:19.769047 5224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:19.769697 5224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:19.770699 5224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:19.770767 5224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:19.770838 5224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:19.770881 5224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:19.780251 5224 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33737
I20260929 16:26:19.780295 5351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33737 every 8 connection(s)
I20260929 16:26:19.781353 5224 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:26:19.787886 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:19.783278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36653 (local address 127.0.98.62:32885)
0929 16:26:19.783450 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:19.783454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:19.784641 (+ 1187us) server_negotiation.cc:407] Connection header received
0929 16:26:19.786045 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:19.786048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:19.786089 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:19.786170 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:19.787336 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:19.787339 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:19.787341 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:19.787353 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:19.787374 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:19.787383 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:19.787474 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:19.787710 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 16:26:19.787758 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:26:19.788827 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:19.783721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32885 (local address 127.0.98.2:36653)
0929 16:26:19.784460 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:19.784501 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:19.785800 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:19.786348 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:19.786355 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:19.786446 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:19.787036 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:26:19.787037 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:19.787095 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:26:19.787169 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:19.787506 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:19.787514 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:19.787519 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 16:26:19.787712 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 16:26:19.787927 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":133,"threads_started":1}
I20260929 16:26:19.790395 5352 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32885
I20260929 16:26:19.790656 5352 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:19.790975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 5224
I20260929 16:26:19.791074 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:26:19.791095 5352 heartbeater.cc:511] Master 127.0.98.62:32885 requested a full tablet report, sending...
I20260929 16:26:19.791368 392 external_mini_cluster.cc:1428] Setting key daa80b0434655b6bc7c67d386ce76346
I20260929 16:26:19.792341 5033 ts_manager.cc:194] Registered new tserver with Master: 285f2041ddf64973a8dee7ca02020305 (127.0.98.2:33737)
I20260929 16:26:19.793726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32885
--builtin_ntp_servers=127.0.98.20:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 16:26:19.910703 5355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:19.911150 5355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:19.911224 5355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:19.915743 5355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:19.915812 5355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:19.915874 5355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:26:19.920810 5355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:19.921989 5355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:19.923386 5355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:19.933812 5363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:19.933864 5361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:19.933864 5360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:19.933918 5355 server_base.cc:1034] running on GCE node
I20260929 16:26:19.934616 5355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:19.935364 5355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:19.936568 5355 hybrid_clock.cc:648] HybridClock initialized: now 1790699179936554 us; error 31 us; skew 500 ppm
I20260929 16:26:19.938712 5355 webserver.cc:460] Webserver started at http://127.0.98.3:35649/ using document root <none> and password file <none>
I20260929 16:26:19.939320 5355 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:19.939414 5355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:19.939711 5355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:19.942128 5355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "041408be3c154c418d0f9b4c267e0a31"
format_stamp: "Formatted at 2026-09-29 16:26:19 on dist-test-slave-vq9v"
server_key: "32b429daa7cf7fe77c386addd558880d"
server_key_iv: "e4c12fd9f01a3fb3e6c9556587733157"
server_key_version: "encryptionkey@0"
I20260929 16:26:19.942729 5355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "041408be3c154c418d0f9b4c267e0a31"
format_stamp: "Formatted at 2026-09-29 16:26:19 on dist-test-slave-vq9v"
server_key: "32b429daa7cf7fe77c386addd558880d"
server_key_iv: "e4c12fd9f01a3fb3e6c9556587733157"
server_key_version: "encryptionkey@0"
I20260929 16:26:19.946583 5355 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:26:19.949163 5369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:19.950501 5355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:26:19.950642 5355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "041408be3c154c418d0f9b4c267e0a31"
format_stamp: "Formatted at 2026-09-29 16:26:19 on dist-test-slave-vq9v"
server_key: "32b429daa7cf7fe77c386addd558880d"
server_key_iv: "e4c12fd9f01a3fb3e6c9556587733157"
server_key_version: "encryptionkey@0"
I20260929 16:26:19.950793 5355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:19.955430 5355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:19.956044 5355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:19.956230 5355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:19.956784 5355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:19.957775 5355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:19.957845 5355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:19.957912 5355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:19.957962 5355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:19.966562 5355 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33351
I20260929 16:26:19.966619 5482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33351 every 8 connection(s)
I20260929 16:26:19.967598 5355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:26:19.970603 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 5355
I20260929 16:26:19.970762 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:26:19.971077 392 external_mini_cluster.cc:1428] Setting key 189e03f08de555cd561240f7ff72a227
I20260929 16:26:19.974798 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:19.969570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41135 (local address 127.0.98.62:32885)
0929 16:26:19.969721 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:19.969725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:19.971073 (+ 1348us) server_negotiation.cc:407] Connection header received
0929 16:26:19.972811 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:19.972814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:19.972855 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:19.972944 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:19.974303 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:19.974306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:19.974308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:19.974320 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:19.974394 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:19.974409 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:19.974507 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:19.974650 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 16:26:19.974689 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:26:19.975878 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:19.970052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32885 (local address 127.0.98.3:41135)
0929 16:26:19.970873 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:19.970934 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:19.972605 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:19.973144 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:19.973224 (+ 80us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:19.973376 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:19.974034 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:26:19.974036 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:19.974078 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:26:19.974158 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:19.974521 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:19.974528 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:19.974533 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 16:26:19.974678 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 16:26:19.974931 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":114,"threads_started":1}
I20260929 16:26:19.977192 5483 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32885
I20260929 16:26:19.977519 5483 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:19.977989 5483 heartbeater.cc:511] Master 127.0.98.62:32885 requested a full tablet report, sending...
I20260929 16:26:19.979140 5032 ts_manager.cc:194] Registered new tserver with Master: 041408be3c154c418d0f9b4c267e0a31 (127.0.98.3:33351)
I20260929 16:26:19.985846 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:26:19.989890 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:19.988104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34118 (local address 127.0.98.62:32885)
0929 16:26:19.988245 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:19.988248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:19.988375 (+ 127us) server_negotiation.cc:407] Connection header received
0929 16:26:19.988507 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:19.988509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:19.988544 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:19.988620 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:19.989563 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:19.989567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:19.989569 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:19.989579 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:19.989597 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:19.989605 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:19.989695 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:19.989746 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 16:26:19.989780 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:26:19.993026 5032 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:26:19.996088 5032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:26:20.008970 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:20.005913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33737 (local address 127.0.0.1:53486)
0929 16:26:20.006321 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:20.006381 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:20.006499 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:20.007012 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:20.007019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:20.007103 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:20.007716 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:26:20.007720 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:20.007744 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:26:20.007821 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:20.008602 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:20.008607 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:20.008614 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 16:26:20.008709 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 16:26:20.008784 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"spinlock_wait_cycles":3328,"thread_start_us":104,"threads_started":1}
I20260929 16:26:20.009014 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:20.006078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53486 (local address 127.0.98.2:33737)
0929 16:26:20.006491 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:20.006504 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:20.006533 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:26:20.006594 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:20.006600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:20.006825 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:20.007410 (+ 585us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:20.008401 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:20.008406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:20.008417 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:20.008426 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:20.008465 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:20.008496 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:20.008599 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:20.008756 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 16:26:20.008841 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":101,"threads_started":1}
I20260929 16:26:20.009573 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:20.005654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37036 (local address 127.0.98.3:33351)
0929 16:26:20.006813 (+ 1159us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:20.006825 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:20.006848 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:26:20.006900 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:20.006906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:20.007089 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:20.007226 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:20.008097 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:20.008108 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:20.008125 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:20.008146 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:20.008193 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:20.008228 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:20.008333 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:20.009328 (+ 995us) server_negotiation.cc:299] Negotiation successful
0929 16:26:20.009405 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1043,"thread_start_us":177,"threads_started":1}
I20260929 16:26:20.009800 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:20.005485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33351 (local address 127.0.0.1:37036)
0929 16:26:20.006321 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:20.006381 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:20.006498 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:20.007374 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:20.007377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:20.007392 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:20.007905 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:26:20.007909 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:20.007921 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:26:20.007953 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:20.008381 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:20.008390 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:20.008395 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 16:26:20.008815 (+ 420us) client_negotiation.cc:240] Negotiation successful
0929 16:26:20.008849 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"spinlock_wait_cycles":7296,"thread_start_us":88,"threads_started":1}
I20260929 16:26:20.009919 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:20.006644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50212 (local address 127.0.98.1:38891)
0929 16:26:20.007003 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:20.007014 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:20.007252 (+ 238us) server_negotiation.cc:407] Connection header received
0929 16:26:20.007974 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:20.007983 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:20.008160 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:20.008313 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:20.009106 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:20.009113 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:20.009126 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:20.009138 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:20.009185 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:20.009256 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:20.009352 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:20.009677 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 16:26:20.009750 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":85,"threads_started":1}
I20260929 16:26:20.010293 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:20.006259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38891 (local address 127.0.0.1:50212)
0929 16:26:20.007157 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:20.007172 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:20.007314 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:20.008297 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:20.008299 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:20.008314 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:20.008928 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:26:20.008929 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:20.008939 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:26:20.008967 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:20.009521 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:20.009524 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:20.009527 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 16:26:20.010128 (+ 601us) client_negotiation.cc:240] Negotiation successful
0929 16:26:20.010169 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"thread_start_us":85,"threads_started":1}
I20260929 16:26:20.012193 5286 tablet_service.cc:1467] Processing CreateTablet for tablet c182428fd2ed4b658f543e9de2eef91b (DEFAULT_TABLE table=test-table [id=d6dfb1cf314a416b8772ca430ff99eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:20.012846 5417 tablet_service.cc:1467] Processing CreateTablet for tablet c182428fd2ed4b658f543e9de2eef91b (DEFAULT_TABLE table=test-table [id=d6dfb1cf314a416b8772ca430ff99eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:20.012844 5155 tablet_service.cc:1467] Processing CreateTablet for tablet c182428fd2ed4b658f543e9de2eef91b (DEFAULT_TABLE table=test-table [id=d6dfb1cf314a416b8772ca430ff99eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:20.013341 5286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c182428fd2ed4b658f543e9de2eef91b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:20.013890 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c182428fd2ed4b658f543e9de2eef91b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:20.013890 5155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c182428fd2ed4b658f543e9de2eef91b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:20.020699 5500 tablet_bootstrap.cc:492] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305: Bootstrap starting.
I20260929 16:26:20.022326 5501 tablet_bootstrap.cc:492] T c182428fd2ed4b658f543e9de2eef91b P 041408be3c154c418d0f9b4c267e0a31: Bootstrap starting.
I20260929 16:26:20.022502 5502 tablet_bootstrap.cc:492] T c182428fd2ed4b658f543e9de2eef91b P 638c6b6277024213a854b9aff6f16c52: Bootstrap starting.
I20260929 16:26:20.023958 5500 tablet_bootstrap.cc:654] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:20.024766 5501 tablet_bootstrap.cc:654] T c182428fd2ed4b658f543e9de2eef91b P 041408be3c154c418d0f9b4c267e0a31: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:20.024766 5502 tablet_bootstrap.cc:654] T c182428fd2ed4b658f543e9de2eef91b P 638c6b6277024213a854b9aff6f16c52: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:20.024879 5500 log.cc:824] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:20.025682 5502 log.cc:824] T c182428fd2ed4b658f543e9de2eef91b P 638c6b6277024213a854b9aff6f16c52: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:20.025686 5501 log.cc:824] T c182428fd2ed4b658f543e9de2eef91b P 041408be3c154c418d0f9b4c267e0a31: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:20.027442 5500 tablet_bootstrap.cc:492] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305: No bootstrap required, opened a new log
I20260929 16:26:20.027699 5500 ts_tablet_manager.cc:1397] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260929 16:26:20.027787 5501 tablet_bootstrap.cc:492] T c182428fd2ed4b658f543e9de2eef91b P 041408be3c154c418d0f9b4c267e0a31: No bootstrap required, opened a new log
I20260929 16:26:20.027981 5501 ts_tablet_manager.cc:1397] T c182428fd2ed4b658f543e9de2eef91b P 041408be3c154c418d0f9b4c267e0a31: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:26:20.028218 5502 tablet_bootstrap.cc:492] T c182428fd2ed4b658f543e9de2eef91b P 638c6b6277024213a854b9aff6f16c52: No bootstrap required, opened a new log
I20260929 16:26:20.028371 5502 ts_tablet_manager.cc:1397] T c182428fd2ed4b658f543e9de2eef91b P 638c6b6277024213a854b9aff6f16c52: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:26:20.031836 5500 raft_consensus.cc:348] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041408be3c154c418d0f9b4c267e0a31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33351 } } peers { permanent_uuid: "638c6b6277024213a854b9aff6f16c52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38891 } } peers { permanent_uuid: "285f2041ddf64973a8dee7ca02020305" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33737 } }
I20260929 16:26:20.032137 5500 raft_consensus.cc:374] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:20.032205 5500 raft_consensus.cc:729] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285f2041ddf64973a8dee7ca02020305, State: Initialized, Role: FOLLOWER
I20260929 16:26:20.032714 5500 consensus_queue.cc:260] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041408be3c154c418d0f9b4c267e0a31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33351 } } peers { permanent_uuid: "638c6b6277024213a854b9aff6f16c52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38891 } } peers { permanent_uuid: "285f2041ddf64973a8dee7ca02020305" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33737 } }
I20260929 16:26:20.032747 5502 raft_consensus.cc:348] T c182428fd2ed4b658f543e9de2eef91b P 638c6b6277024213a854b9aff6f16c52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041408be3c154c418d0f9b4c267e0a31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33351 } } peers { permanent_uuid: "638c6b6277024213a854b9aff6f16c52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38891 } } peers { permanent_uuid: "285f2041ddf64973a8dee7ca02020305" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33737 } }
I20260929 16:26:20.032747 5501 raft_consensus.cc:348] T c182428fd2ed4b658f543e9de2eef91b P 041408be3c154c418d0f9b4c267e0a31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041408be3c154c418d0f9b4c267e0a31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33351 } } peers { permanent_uuid: "638c6b6277024213a854b9aff6f16c52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38891 } } peers { permanent_uuid: "285f2041ddf64973a8dee7ca02020305" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33737 } }
I20260929 16:26:20.033008 5502 raft_consensus.cc:374] T c182428fd2ed4b658f543e9de2eef91b P 638c6b6277024213a854b9aff6f16c52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:20.033008 5501 raft_consensus.cc:374] T c182428fd2ed4b658f543e9de2eef91b P 041408be3c154c418d0f9b4c267e0a31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:20.033066 5501 raft_consensus.cc:729] T c182428fd2ed4b658f543e9de2eef91b P 041408be3c154c418d0f9b4c267e0a31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 041408be3c154c418d0f9b4c267e0a31, State: Initialized, Role: FOLLOWER
I20260929 16:26:20.033066 5502 raft_consensus.cc:729] T c182428fd2ed4b658f543e9de2eef91b P 638c6b6277024213a854b9aff6f16c52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 638c6b6277024213a854b9aff6f16c52, State: Initialized, Role: FOLLOWER
I20260929 16:26:20.033565 5352 heartbeater.cc:503] Master 127.0.98.62:32885 was elected leader, sending a full tablet report...
I20260929 16:26:20.033555 5502 consensus_queue.cc:260] T c182428fd2ed4b658f543e9de2eef91b P 638c6b6277024213a854b9aff6f16c52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041408be3c154c418d0f9b4c267e0a31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33351 } } peers { permanent_uuid: "638c6b6277024213a854b9aff6f16c52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38891 } } peers { permanent_uuid: "285f2041ddf64973a8dee7ca02020305" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33737 } }
I20260929 16:26:20.033932 5500 ts_tablet_manager.cc:1428] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260929 16:26:20.034294 5221 heartbeater.cc:503] Master 127.0.98.62:32885 was elected leader, sending a full tablet report...
I20260929 16:26:20.034546 5502 ts_tablet_manager.cc:1428] T c182428fd2ed4b658f543e9de2eef91b P 638c6b6277024213a854b9aff6f16c52: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:26:20.034837 5501 consensus_queue.cc:260] T c182428fd2ed4b658f543e9de2eef91b P 041408be3c154c418d0f9b4c267e0a31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041408be3c154c418d0f9b4c267e0a31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33351 } } peers { permanent_uuid: "638c6b6277024213a854b9aff6f16c52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38891 } } peers { permanent_uuid: "285f2041ddf64973a8dee7ca02020305" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33737 } }
I20260929 16:26:20.035928 5501 ts_tablet_manager.cc:1428] T c182428fd2ed4b658f543e9de2eef91b P 041408be3c154c418d0f9b4c267e0a31: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260929 16:26:20.036284 5483 heartbeater.cc:503] Master 127.0.98.62:32885 was elected leader, sending a full tablet report...
I20260929 16:26:20.084398 5506 raft_consensus.cc:482] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:26:20.084720 5506 raft_consensus.cc:504] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041408be3c154c418d0f9b4c267e0a31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33351 } } peers { permanent_uuid: "638c6b6277024213a854b9aff6f16c52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38891 } } peers { permanent_uuid: "285f2041ddf64973a8dee7ca02020305" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33737 } }
I20260929 16:26:20.085912 5506 leader_election.cc:288] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 041408be3c154c418d0f9b4c267e0a31 (127.0.98.3:33351), 638c6b6277024213a854b9aff6f16c52 (127.0.98.1:38891)
I20260929 16:26:20.088272 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:20.086329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33351 (local address 127.0.98.2:60183)
0929 16:26:20.086483 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:20.086502 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:20.086619 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:20.087107 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:20.087114 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:20.087132 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:20.087683 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:26:20.087685 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:20.087705 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:26:20.087732 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:20.088006 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:20.088010 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:20.088013 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 16:26:20.088096 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 16:26:20.088136 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 16:26:20.088272 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:20.086385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60183 (local address 127.0.98.3:33351)
0929 16:26:20.086561 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:20.086569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:20.086589 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:26:20.086909 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:20.086913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:20.086961 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:20.087082 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:20.087857 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:20.087861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:20.087864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:20.087875 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:20.087894 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:20.087905 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:20.088019 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:20.088108 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 16:26:20.088147 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:26:20.088585 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:20.086785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50659 (local address 127.0.98.1:38891)
0929 16:26:20.086945 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:20.086950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:20.087189 (+ 239us) server_negotiation.cc:407] Connection header received
0929 16:26:20.087287 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:20.087290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:20.087358 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:20.087446 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:20.088141 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:20.088144 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:20.088146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:20.088157 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:20.088176 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:20.088184 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:20.088285 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:20.088402 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 16:26:20.088450 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:26:20.088907 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:20.086646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38891 (local address 127.0.98.2:50659)
0929 16:26:20.087066 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:20.087080 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:20.087174 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:20.087493 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:20.087497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:20.087515 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:20.087946 (+ 431us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:26:20.087948 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:20.087959 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:26:20.087991 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:20.088281 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:20.088284 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:20.088287 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 16:26:20.088741 (+ 454us) client_negotiation.cc:240] Negotiation successful
0929 16:26:20.088780 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":172,"threads_started":1}
I20260929 16:26:20.088948 5437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c182428fd2ed4b658f543e9de2eef91b" candidate_uuid: "285f2041ddf64973a8dee7ca02020305" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "041408be3c154c418d0f9b4c267e0a31" is_pre_election: true
I20260929 16:26:20.089357 5437 raft_consensus.cc:2454] T c182428fd2ed4b658f543e9de2eef91b P 041408be3c154c418d0f9b4c267e0a31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 285f2041ddf64973a8dee7ca02020305 in term 0.
I20260929 16:26:20.089713 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c182428fd2ed4b658f543e9de2eef91b" candidate_uuid: "285f2041ddf64973a8dee7ca02020305" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "638c6b6277024213a854b9aff6f16c52" is_pre_election: true
I20260929 16:26:20.089926 5241 leader_election.cc:302] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 041408be3c154c418d0f9b4c267e0a31, 285f2041ddf64973a8dee7ca02020305; no voters:
I20260929 16:26:20.089988 5175 raft_consensus.cc:2454] T c182428fd2ed4b658f543e9de2eef91b P 638c6b6277024213a854b9aff6f16c52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 285f2041ddf64973a8dee7ca02020305 in term 0.
I20260929 16:26:20.090233 5506 raft_consensus.cc:2781] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:26:20.090339 5506 raft_consensus.cc:482] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:26:20.090389 5506 raft_consensus.cc:3037] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:20.091445 5506 raft_consensus.cc:504] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041408be3c154c418d0f9b4c267e0a31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33351 } } peers { permanent_uuid: "638c6b6277024213a854b9aff6f16c52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38891 } } peers { permanent_uuid: "285f2041ddf64973a8dee7ca02020305" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33737 } }
I20260929 16:26:20.091823 5506 leader_election.cc:288] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305 [CANDIDATE]: Term 1 election: Requested vote from peers 041408be3c154c418d0f9b4c267e0a31 (127.0.98.3:33351), 638c6b6277024213a854b9aff6f16c52 (127.0.98.1:38891)
I20260929 16:26:20.092393 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c182428fd2ed4b658f543e9de2eef91b" candidate_uuid: "285f2041ddf64973a8dee7ca02020305" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "638c6b6277024213a854b9aff6f16c52"
I20260929 16:26:20.092371 5437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c182428fd2ed4b658f543e9de2eef91b" candidate_uuid: "285f2041ddf64973a8dee7ca02020305" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "041408be3c154c418d0f9b4c267e0a31"
I20260929 16:26:20.092502 5175 raft_consensus.cc:3037] T c182428fd2ed4b658f543e9de2eef91b P 638c6b6277024213a854b9aff6f16c52 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:20.092600 5437 raft_consensus.cc:3037] T c182428fd2ed4b658f543e9de2eef91b P 041408be3c154c418d0f9b4c267e0a31 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:20.093691 5175 raft_consensus.cc:2454] T c182428fd2ed4b658f543e9de2eef91b P 638c6b6277024213a854b9aff6f16c52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285f2041ddf64973a8dee7ca02020305 in term 1.
I20260929 16:26:20.093892 5437 raft_consensus.cc:2454] T c182428fd2ed4b658f543e9de2eef91b P 041408be3c154c418d0f9b4c267e0a31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 285f2041ddf64973a8dee7ca02020305 in term 1.
I20260929 16:26:20.093993 5241 leader_election.cc:302] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 285f2041ddf64973a8dee7ca02020305, 638c6b6277024213a854b9aff6f16c52; no voters:
I20260929 16:26:20.094252 5506 raft_consensus.cc:2781] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:20.094553 5506 raft_consensus.cc:686] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305 [term 1 LEADER]: Becoming Leader. State: Replica: 285f2041ddf64973a8dee7ca02020305, State: Running, Role: LEADER
I20260929 16:26:20.094937 5506 consensus_queue.cc:237] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041408be3c154c418d0f9b4c267e0a31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33351 } } peers { permanent_uuid: "638c6b6277024213a854b9aff6f16c52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38891 } } peers { permanent_uuid: "285f2041ddf64973a8dee7ca02020305" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33737 } }
W20260929 16:26:20.097266 5222 tablet.cc:2369] T c182428fd2ed4b658f543e9de2eef91b P 638c6b6277024213a854b9aff6f16c52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:26:20.098488 5032 catalog_manager.cc:5495] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305 reported cstate change: term changed from 0 to 1, leader changed from <none> to 285f2041ddf64973a8dee7ca02020305 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "285f2041ddf64973a8dee7ca02020305" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "041408be3c154c418d0f9b4c267e0a31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "638c6b6277024213a854b9aff6f16c52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "285f2041ddf64973a8dee7ca02020305" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260929 16:26:20.110865 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:20.109289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53496 (local address 127.0.98.2:33737)
0929 16:26:20.109420 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:20.109423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:20.109522 (+ 99us) server_negotiation.cc:407] Connection header received
0929 16:26:20.109660 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:20.109663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:20.109724 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:20.109796 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:20.110446 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:20.110449 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:20.110451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:20.110462 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:20.110482 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:20.110490 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:20.110587 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:20.110632 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 16:26:20.110667 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:26:20.116838 5437 raft_consensus.cc:1261] T c182428fd2ed4b658f543e9de2eef91b P 041408be3c154c418d0f9b4c267e0a31 [term 1 FOLLOWER]: Refusing update from remote peer 285f2041ddf64973a8dee7ca02020305: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:20.116838 5175 raft_consensus.cc:1261] T c182428fd2ed4b658f543e9de2eef91b P 638c6b6277024213a854b9aff6f16c52 [term 1 FOLLOWER]: Refusing update from remote peer 285f2041ddf64973a8dee7ca02020305: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:20.117641 5506 consensus_queue.cc:1035] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305 [LEADER]: Connected to new peer: Peer: permanent_uuid: "041408be3c154c418d0f9b4c267e0a31" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:20.117900 5510 consensus_queue.cc:1035] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305 [LEADER]: Connected to new peer: Peer: permanent_uuid: "638c6b6277024213a854b9aff6f16c52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:20.134764 5033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:20.134934 5033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:20.137578 5033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76ae60cb45af468abf8f04fa21d0e83d: Sending DeleteTablet for 3 replicas of tablet c182428fd2ed4b658f543e9de2eef91b
I20260929 16:26:20.138526 5286 tablet_service.cc:1514] Processing DeleteTablet for tablet c182428fd2ed4b658f543e9de2eef91b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:20 UTC) from {username='slave'} at 127.0.0.1:53486
I20260929 16:26:20.138566 5155 tablet_service.cc:1514] Processing DeleteTablet for tablet c182428fd2ed4b658f543e9de2eef91b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:20 UTC) from {username='slave'} at 127.0.0.1:50212
I20260929 16:26:20.138598 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet c182428fd2ed4b658f543e9de2eef91b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:20 UTC) from {username='slave'} at 127.0.0.1:37036
I20260929 16:26:20.139091 5524 tablet_replica.cc:330] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305: stopping tablet replica
I20260929 16:26:20.139146 5525 tablet_replica.cc:330] T c182428fd2ed4b658f543e9de2eef91b P 041408be3c154c418d0f9b4c267e0a31: stopping tablet replica
I20260929 16:26:20.139421 5523 tablet_replica.cc:330] T c182428fd2ed4b658f543e9de2eef91b P 638c6b6277024213a854b9aff6f16c52: stopping tablet replica
I20260929 16:26:20.139464 5525 raft_consensus.cc:2229] T c182428fd2ed4b658f543e9de2eef91b P 041408be3c154c418d0f9b4c267e0a31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:20.139755 5523 raft_consensus.cc:2229] T c182428fd2ed4b658f543e9de2eef91b P 638c6b6277024213a854b9aff6f16c52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:20.139817 5525 raft_consensus.cc:2258] T c182428fd2ed4b658f543e9de2eef91b P 041408be3c154c418d0f9b4c267e0a31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:20.140044 5523 raft_consensus.cc:2258] T c182428fd2ed4b658f543e9de2eef91b P 638c6b6277024213a854b9aff6f16c52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:20.140312 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 5093
I20260929 16:26:20.140437 5524 raft_consensus.cc:2229] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:26:20.140861 5524 raft_consensus.cc:2258] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:20.141760 5525 ts_tablet_manager.cc:1905] T c182428fd2ed4b658f543e9de2eef91b P 041408be3c154c418d0f9b4c267e0a31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:20.142776 5524 ts_tablet_manager.cc:1905] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:20.144533 5525 ts_tablet_manager.cc:1918] T c182428fd2ed4b658f543e9de2eef91b P 041408be3c154c418d0f9b4c267e0a31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:26:20.144626 5525 log.cc:1196] T c182428fd2ed4b658f543e9de2eef91b P 041408be3c154c418d0f9b4c267e0a31: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/c182428fd2ed4b658f543e9de2eef91b
I20260929 16:26:20.144943 5525 ts_tablet_manager.cc:1939] T c182428fd2ed4b658f543e9de2eef91b P 041408be3c154c418d0f9b4c267e0a31: Deleting consensus metadata
I20260929 16:26:20.145565 5524 ts_tablet_manager.cc:1918] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:26:20.145699 5524 log.cc:1196] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/c182428fd2ed4b658f543e9de2eef91b
I20260929 16:26:20.146021 5524 ts_tablet_manager.cc:1939] T c182428fd2ed4b658f543e9de2eef91b P 285f2041ddf64973a8dee7ca02020305: Deleting consensus metadata
I20260929 16:26:20.147145 5017 catalog_manager.cc:4841] TS 285f2041ddf64973a8dee7ca02020305 (127.0.98.2:33737): tablet c182428fd2ed4b658f543e9de2eef91b (table test-table [id=d6dfb1cf314a416b8772ca430ff99eb7]) successfully deleted
I20260929 16:26:20.147627 5019 catalog_manager.cc:4841] TS 041408be3c154c418d0f9b4c267e0a31 (127.0.98.3:33351): tablet c182428fd2ed4b658f543e9de2eef91b (table test-table [id=d6dfb1cf314a416b8772ca430ff99eb7]) successfully deleted
W20260929 16:26:20.150307 5019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38891 (error 108)
I20260929 16:26:20.151217 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 5224
I20260929 16:26:20.151578 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:20.151174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38891 (local address 127.0.0.1:50216)
0929 16:26:20.151393 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:20.151474 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260929 16:26:20.152048 5019 catalog_manager.cc:4584] TS 638c6b6277024213a854b9aff6f16c52 (127.0.98.1:38891): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c182428fd2ed4b658f543e9de2eef91b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38891: connect: Connection refused (error 111)
I20260929 16:26:20.156783 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 5355
I20260929 16:26:20.162441 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 5002
2026-09-29T16:26:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1008 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 16:26:20.184042 392 mini_oidc.cc:152] Starting JWKS server
I20260929 16:26:20.185927 392 webserver.cc:460] Webserver started at https://127.0.0.1:40559/ using document root <none> and password file <none>
I20260929 16:26:20.186084 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 16:26:20.186374 392 webserver.cc:460] Webserver started at http://127.0.0.1:45621/ using document root <none> and password file <none>
2026-09-29T16:26:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:26:20Z Disabled control of system clock
I20260929 16:26:20.195488 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38877
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38877
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40559/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699152396020-392-0/testchainca.cert with env {}
W20260929 16:26:20.307963 5532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:20.308346 5532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:20.308420 5532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:20.312757 5532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:26:20.312884 5532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:20.312912 5532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:20.312944 5532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:26:20.313004 5532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:26:20.313043 5532 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:26:20.313081 5532 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40559/jwks.json
I20260929 16:26:20.318910 5532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38877
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40559/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699152396020-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:20.320103 5532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:20.321173 5532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:20.330421 5537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:20.330399 5540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:20.330456 5532 server_base.cc:1034] running on GCE node
W20260929 16:26:20.330403 5538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:20.331380 5532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:20.332592 5532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:20.333820 5532 hybrid_clock.cc:648] HybridClock initialized: now 1790699180333806 us; error 28 us; skew 500 ppm
I20260929 16:26:20.335896 5532 webserver.cc:460] Webserver started at http://127.0.98.62:34227/ using document root <none> and password file <none>
I20260929 16:26:20.336517 5532 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:20.336615 5532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:20.336849 5532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:20.339056 5532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "5defc9003e954e7fb9fc4be947566ed4"
format_stamp: "Formatted at 2026-09-29 16:26:20 on dist-test-slave-vq9v"
server_key: "08d0a31282a52c69cd67e27a21bb3bd9"
server_key_iv: "49dd51d7a23c87805d0b3714e2f682f4"
server_key_version: "encryptionkey@0"
I20260929 16:26:20.339664 5532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "5defc9003e954e7fb9fc4be947566ed4"
format_stamp: "Formatted at 2026-09-29 16:26:20 on dist-test-slave-vq9v"
server_key: "08d0a31282a52c69cd67e27a21bb3bd9"
server_key_iv: "49dd51d7a23c87805d0b3714e2f682f4"
server_key_version: "encryptionkey@0"
I20260929 16:26:20.343676 5532 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:26:20.346366 5546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:20.347694 5532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:20.347949 5532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "5defc9003e954e7fb9fc4be947566ed4"
format_stamp: "Formatted at 2026-09-29 16:26:20 on dist-test-slave-vq9v"
server_key: "08d0a31282a52c69cd67e27a21bb3bd9"
server_key_iv: "49dd51d7a23c87805d0b3714e2f682f4"
server_key_version: "encryptionkey@0"
I20260929 16:26:20.348078 5532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:20.360996 5532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:20.361831 5532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:20.362047 5532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:20.368737 5532 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38877
I20260929 16:26:20.368752 5598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38877 every 8 connection(s)
I20260929 16:26:20.369798 5532 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:26:20.371446 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 5532
I20260929 16:26:20.371588 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:26:20.371894 392 external_mini_cluster.cc:1428] Setting key 22fa8938a88f0643e74dc8500b9111f3
I20260929 16:26:20.373492 5599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:20.379241 5599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4: Bootstrap starting.
I20260929 16:26:20.381603 5599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:20.381709 5602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:20.373378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55426 (local address 127.0.98.62:38877)
0929 16:26:20.374026 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:20.374042 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:20.374089 (+ 47us) server_negotiation.cc:407] Connection header received
0929 16:26:20.374795 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:20.374813 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:20.375256 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:20.375633 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:20.376075 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:20.377070 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:20.378015 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:20.378501 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:20.378693 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:20.379758 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:20.379762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:20.379775 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:20.380187 (+ 412us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:20.380246 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:20.380268 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:20.380382 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:20.380549 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 16:26:20.380880 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":133,"threads_started":1}
I20260929 16:26:20.382392 5599 log.cc:824] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:20.384649 5599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4: No bootstrap required, opened a new log
I20260929 16:26:20.384806 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:26:20.388679 5599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5defc9003e954e7fb9fc4be947566ed4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38877 } }
I20260929 16:26:20.388922 5599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:20.388964 5599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5defc9003e954e7fb9fc4be947566ed4, State: Initialized, Role: FOLLOWER
I20260929 16:26:20.389424 5599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5defc9003e954e7fb9fc4be947566ed4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38877 } }
I20260929 16:26:20.389554 5599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:26:20.389597 5599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:26:20.389669 5599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:20.390583 5599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5defc9003e954e7fb9fc4be947566ed4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38877 } }
I20260929 16:26:20.390894 5599 leader_election.cc:302] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5defc9003e954e7fb9fc4be947566ed4; no voters:
I20260929 16:26:20.391173 5599 leader_election.cc:288] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:26:20.391310 5604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:20.391566 5604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4 [term 1 LEADER]: Becoming Leader. State: Replica: 5defc9003e954e7fb9fc4be947566ed4, State: Running, Role: LEADER
I20260929 16:26:20.391948 5604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5defc9003e954e7fb9fc4be947566ed4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38877 } }
I20260929 16:26:20.392547 5599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:26:20.393728 5605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5defc9003e954e7fb9fc4be947566ed4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5defc9003e954e7fb9fc4be947566ed4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38877 } } }
I20260929 16:26:20.393857 5605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:20.394198 5606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5defc9003e954e7fb9fc4be947566ed4. Latest consensus state: current_term: 1 leader_uuid: "5defc9003e954e7fb9fc4be947566ed4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5defc9003e954e7fb9fc4be947566ed4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38877 } } }
I20260929 16:26:20.394317 5606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:20.394599 5613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:26:20.397568 5613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:26:20.403357 5613 catalog_manager.cc:1347] Generated new cluster ID: 45aca57bdded47c9908a41278d6b5426
I20260929 16:26:20.403450 5613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:26:20.418551 5613 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:26:20.419900 5613 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:26:20.433282 5613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5defc9003e954e7fb9fc4be947566ed4: Generated new TSK 0
I20260929 16:26:20.434227 5613 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 16:26:20.473946 5629 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 16:26:20.474071 5629 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 16:26:20.474572 5602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:20.449643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.0.98.62:38877)
0929 16:26:20.449813 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:20.449819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:20.450104 (+ 285us) server_negotiation.cc:407] Connection header received
0929 16:26:20.450336 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:20.450340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:20.450397 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:20.450498 (+ 101us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:26:20.450814 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:20.451493 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:20.452805 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:20.453082 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:20.453252 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:20.453642 (+ 390us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:26:20.474124 (+ 20482us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 16:26:20.474452 (+ 328us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 16:26:20.474676 5602 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 16:26:20.476024 5628 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:20.449582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38877 (local address 127.0.0.1:55436)
0929 16:26:20.449976 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:20.449993 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:20.450138 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:20.450538 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:20.450542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:20.450555 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:26:20.450697 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:20.450703 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:20.451617 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:20.451622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:20.452676 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:20.452689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:20.453234 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:20.453259 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:20.453357 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:20.453416 (+ 59us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:26:20.474436 (+ 21020us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 16:26:20.474700 (+ 264us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38877: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":140,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:38877: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 16:26:20.476806 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 5532
2026-09-29T16:26:20Z chronyd exiting
I20260929 16:26:20.514415 392 test_util.cc:175] -----------------------------------------------
I20260929 16:26:20.514504 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790699152396020-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (331 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 16:26:20.515833 392 mini_oidc.cc:152] Starting JWKS server
I20260929 16:26:20.516767 392 webserver.cc:460] Webserver started at https://127.0.0.1:38413/ using document root <none> and password file <none>
I20260929 16:26:20.516855 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 16:26:20.517246 392 webserver.cc:460] Webserver started at http://127.0.0.1:40853/ using document root <none> and password file <none>
2026-09-29T16:26:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:26:20Z Disabled control of system clock
I20260929 16:26:20.526240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43085
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43085
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38413/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699152396020-392-0/testchainca.cert with env {}
W20260929 16:26:20.638059 5635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:20.638365 5635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:20.638406 5635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:20.642513 5635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:26:20.642575 5635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:20.642591 5635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:20.642609 5635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:26:20.642633 5635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:26:20.642650 5635 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:26:20.642664 5635 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38413/jwks.json
I20260929 16:26:20.648209 5635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43085
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38413/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699152396020-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:20.649323 5635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:20.650313 5635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:20.659039 5641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:20.659075 5640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:20.659219 5635 server_base.cc:1034] running on GCE node
W20260929 16:26:20.659114 5643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:20.659783 5635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:20.660954 5635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:20.662127 5635 hybrid_clock.cc:648] HybridClock initialized: now 1790699180662099 us; error 39 us; skew 500 ppm
I20260929 16:26:20.664007 5635 webserver.cc:460] Webserver started at http://127.0.98.62:42801/ using document root <none> and password file <none>
I20260929 16:26:20.664651 5635 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:20.664714 5635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:20.664953 5635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:20.667101 5635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "477d622637064743bfaa3a4487ddafe5"
format_stamp: "Formatted at 2026-09-29 16:26:20 on dist-test-slave-vq9v"
server_key: "1dc823f4d7281b203699c758c6a3d44d"
server_key_iv: "5e8780398735e2f6f6767f2144c49fdc"
server_key_version: "encryptionkey@0"
I20260929 16:26:20.667665 5635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "477d622637064743bfaa3a4487ddafe5"
format_stamp: "Formatted at 2026-09-29 16:26:20 on dist-test-slave-vq9v"
server_key: "1dc823f4d7281b203699c758c6a3d44d"
server_key_iv: "5e8780398735e2f6f6767f2144c49fdc"
server_key_version: "encryptionkey@0"
I20260929 16:26:20.671573 5635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:26:20.674654 5649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:20.675983 5635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:26:20.676186 5635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "477d622637064743bfaa3a4487ddafe5"
format_stamp: "Formatted at 2026-09-29 16:26:20 on dist-test-slave-vq9v"
server_key: "1dc823f4d7281b203699c758c6a3d44d"
server_key_iv: "5e8780398735e2f6f6767f2144c49fdc"
server_key_version: "encryptionkey@0"
I20260929 16:26:20.676311 5635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:20.702814 5635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:20.703616 5635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:20.703822 5635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:20.711273 5635 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43085
I20260929 16:26:20.711285 5701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43085 every 8 connection(s)
I20260929 16:26:20.712399 5635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:26:20.715603 5702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:20.721927 5702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5: Bootstrap starting.
I20260929 16:26:20.722445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 5635
I20260929 16:26:20.722600 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:26:20.722889 392 external_mini_cluster.cc:1428] Setting key 37e209defd02310a1cb3ed72ec89fe67
I20260929 16:26:20.724831 5702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:20.725709 5702 log.cc:824] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:20.727952 5702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5: No bootstrap required, opened a new log
I20260929 16:26:20.732461 5702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477d622637064743bfaa3a4487ddafe5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43085 } }
I20260929 16:26:20.732751 5702 raft_consensus.cc:374] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:20.732838 5702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 477d622637064743bfaa3a4487ddafe5, State: Initialized, Role: FOLLOWER
I20260929 16:26:20.732980 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:20.724388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45502 (local address 127.0.98.62:43085)
0929 16:26:20.725102 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:20.725118 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:20.725157 (+ 39us) server_negotiation.cc:407] Connection header received
0929 16:26:20.725953 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:20.725974 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:20.726426 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:20.726740 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:20.727083 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:20.728089 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:20.729338 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:20.729887 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:20.730015 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:20.730892 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:20.730898 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:20.730913 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:20.731348 (+ 435us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:20.731409 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:20.731443 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:20.731521 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:20.731726 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 16:26:20.732066 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":147,"threads_started":1}
I20260929 16:26:20.733472 5702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477d622637064743bfaa3a4487ddafe5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43085 } }
I20260929 16:26:20.733656 5702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:26:20.733713 5702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:26:20.733811 5702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:20.734826 5702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477d622637064743bfaa3a4487ddafe5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43085 } }
I20260929 16:26:20.735241 5702 leader_election.cc:302] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 477d622637064743bfaa3a4487ddafe5; no voters:
I20260929 16:26:20.735613 5702 leader_election.cc:288] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:26:20.735749 5707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:20.735961 5707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5 [term 1 LEADER]: Becoming Leader. State: Replica: 477d622637064743bfaa3a4487ddafe5, State: Running, Role: LEADER
I20260929 16:26:20.736354 5707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477d622637064743bfaa3a4487ddafe5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43085 } }
I20260929 16:26:20.736522 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:26:20.737035 5702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:26:20.738061 5708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "477d622637064743bfaa3a4487ddafe5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477d622637064743bfaa3a4487ddafe5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43085 } } }
I20260929 16:26:20.738202 5708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:20.738456 5708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 477d622637064743bfaa3a4487ddafe5. Latest consensus state: current_term: 1 leader_uuid: "477d622637064743bfaa3a4487ddafe5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477d622637064743bfaa3a4487ddafe5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43085 } } }
I20260929 16:26:20.738549 5708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:20.738837 5716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:26:20.741937 5716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:26:20.747994 5716 catalog_manager.cc:1347] Generated new cluster ID: 3b09483e2db44e08beeb096628944d76
I20260929 16:26:20.748100 5716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:26:20.769130 5716 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:26:20.770509 5716 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:26:20.780490 5716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 477d622637064743bfaa3a4487ddafe5: Generated new TSK 0
I20260929 16:26:20.781364 5716 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 16:26:20.821225 5732 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 16:26:20.821326 5732 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 16:26:20.822033 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:20.798265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45512 (local address 127.0.98.62:43085)
0929 16:26:20.798401 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:20.798406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:20.798623 (+ 217us) server_negotiation.cc:407] Connection header received
0929 16:26:20.798769 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:20.798773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:20.798834 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:20.798936 (+ 102us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:26:20.799245 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:20.799797 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:20.801078 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:20.801450 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:20.801538 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:20.801917 (+ 379us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:26:20.821676 (+ 19759us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38413/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 16:26:20.821922 (+ 246us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45512: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38413/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 16:26:20.822131 5705 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45512: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38413/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 16:26:20.822279 5731 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:20.798111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43085 (local address 127.0.0.1:45512)
0929 16:26:20.798467 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:20.798484 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:20.798627 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:20.798971 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:20.798974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:20.798983 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:26:20.799116 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:20.799124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:20.799920 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:20.799924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:20.800927 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:20.800939 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:20.801569 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:20.801593 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:20.801696 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:20.801729 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:26:20.821896 (+ 20167us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38413/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 16:26:20.822106 (+ 210us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43085: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38413/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":104,"threads_started":1}
I20260929 16:26:20.823055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 5635
2026-09-29T16:26:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (346 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 16:26:20.861721 392 mini_oidc.cc:152] Starting JWKS server
I20260929 16:26:20.862557 392 webserver.cc:460] Webserver started at https://127.0.0.1:40907/ using document root <none> and password file <none>
I20260929 16:26:20.862648 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 16:26:20.862854 392 webserver.cc:460] Webserver started at http://127.0.0.1:36115/ using document root <none> and password file <none>
2026-09-29T16:26:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:26:20Z Disabled control of system clock
I20260929 16:26:20.872452 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43091
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43091
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40907/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 16:26:20.987587 5738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:20.987847 5738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:20.987886 5738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:20.991869 5738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:26:20.991925 5738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:20.991940 5738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:20.991956 5738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:26:20.991971 5738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:26:20.991987 5738 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:26:20.992002 5738 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40907/jwks.json
I20260929 16:26:20.996625 5738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43091
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40907/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:20.997725 5738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:20.999096 5738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:21.007655 5744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:21.007752 5738 server_base.cc:1034] running on GCE node
W20260929 16:26:21.007773 5743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:21.007766 5746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:21.008538 5738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:21.009709 5738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:21.010913 5738 hybrid_clock.cc:648] HybridClock initialized: now 1790699181010893 us; error 41 us; skew 500 ppm
I20260929 16:26:21.012902 5738 webserver.cc:460] Webserver started at http://127.0.98.62:41459/ using document root <none> and password file <none>
I20260929 16:26:21.013551 5738 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:21.013646 5738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:21.013875 5738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:21.015918 5738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "819fb6d4ff2242879aa26a7de523d50a"
format_stamp: "Formatted at 2026-09-29 16:26:21 on dist-test-slave-vq9v"
server_key: "5f180575883bb545d62cabc7970d8e11"
server_key_iv: "747f87976bc8c5527fca3c73c19cf8da"
server_key_version: "encryptionkey@0"
I20260929 16:26:21.016494 5738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "819fb6d4ff2242879aa26a7de523d50a"
format_stamp: "Formatted at 2026-09-29 16:26:21 on dist-test-slave-vq9v"
server_key: "5f180575883bb545d62cabc7970d8e11"
server_key_iv: "747f87976bc8c5527fca3c73c19cf8da"
server_key_version: "encryptionkey@0"
I20260929 16:26:21.020519 5738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:26:21.023214 5752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:21.024475 5738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 16:26:21.024623 5738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "819fb6d4ff2242879aa26a7de523d50a"
format_stamp: "Formatted at 2026-09-29 16:26:21 on dist-test-slave-vq9v"
server_key: "5f180575883bb545d62cabc7970d8e11"
server_key_iv: "747f87976bc8c5527fca3c73c19cf8da"
server_key_version: "encryptionkey@0"
I20260929 16:26:21.024745 5738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:21.055966 5738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:21.056799 5738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:21.057041 5738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:21.065299 5738 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43091
I20260929 16:26:21.065320 5804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43091 every 8 connection(s)
I20260929 16:26:21.066345 5738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:26:21.069113 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 5738
I20260929 16:26:21.069284 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:26:21.069633 392 external_mini_cluster.cc:1428] Setting key 75322f5fa2119f6ffc0681edbd27a43b
I20260929 16:26:21.069660 5805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:21.077136 5805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a: Bootstrap starting.
I20260929 16:26:21.079816 5805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:21.080458 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:21.071088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36414 (local address 127.0.98.62:43091)
0929 16:26:21.071824 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:21.071843 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:21.071889 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:26:21.072595 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:21.072617 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:21.073144 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:21.073559 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:21.074042 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:21.075468 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:21.076487 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:21.077227 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:21.077386 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:21.078335 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:21.078342 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:21.078359 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:21.078866 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:21.078933 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:21.078959 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:21.079059 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:21.079208 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 16:26:21.079474 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":130,"threads_started":1}
I20260929 16:26:21.080688 5805 log.cc:824] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:21.082792 5805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a: No bootstrap required, opened a new log
I20260929 16:26:21.083416 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:26:21.086647 5805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819fb6d4ff2242879aa26a7de523d50a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43091 } }
I20260929 16:26:21.086889 5805 raft_consensus.cc:374] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:21.086973 5805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819fb6d4ff2242879aa26a7de523d50a, State: Initialized, Role: FOLLOWER
I20260929 16:26:21.087450 5805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819fb6d4ff2242879aa26a7de523d50a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43091 } }
I20260929 16:26:21.087576 5805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:26:21.087664 5805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:26:21.087760 5805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:21.088722 5805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819fb6d4ff2242879aa26a7de523d50a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43091 } }
I20260929 16:26:21.089072 5805 leader_election.cc:302] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 819fb6d4ff2242879aa26a7de523d50a; no voters:
I20260929 16:26:21.089480 5805 leader_election.cc:288] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:26:21.089619 5810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:21.089893 5810 raft_consensus.cc:686] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a [term 1 LEADER]: Becoming Leader. State: Replica: 819fb6d4ff2242879aa26a7de523d50a, State: Running, Role: LEADER
I20260929 16:26:21.090219 5810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819fb6d4ff2242879aa26a7de523d50a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43091 } }
I20260929 16:26:21.090783 5805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:26:21.092049 5811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "819fb6d4ff2242879aa26a7de523d50a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819fb6d4ff2242879aa26a7de523d50a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43091 } } }
I20260929 16:26:21.092178 5811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:21.092448 5812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 819fb6d4ff2242879aa26a7de523d50a. Latest consensus state: current_term: 1 leader_uuid: "819fb6d4ff2242879aa26a7de523d50a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819fb6d4ff2242879aa26a7de523d50a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43091 } } }
I20260929 16:26:21.092546 5812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:21.092659 5819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:26:21.095860 5819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:26:21.101975 5819 catalog_manager.cc:1347] Generated new cluster ID: c3d8f4999dcb4444b3198fbbd3389942
I20260929 16:26:21.102062 5819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:26:21.118161 5819 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:26:21.119521 5819 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:26:21.126197 5819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 819fb6d4ff2242879aa26a7de523d50a: Generated new TSK 0
I20260929 16:26:21.127085 5819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:26:21.144096 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:21.138698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36428 (local address 127.0.98.62:43091)
0929 16:26:21.138838 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:21.138843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:21.139002 (+ 159us) server_negotiation.cc:407] Connection header received
0929 16:26:21.139195 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:21.139198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:21.139251 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:21.139349 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:26:21.139669 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:21.140249 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:21.141531 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:21.141751 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:21.141830 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:21.142227 (+ 397us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:26:21.143713 (+ 1486us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40907/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 16:26:21.143942 (+ 229us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36428: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40907/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 16:26:21.144172 5808 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36428: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40907/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 16:26:21.146016 5834 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:21.138529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43091 (local address 127.0.0.1:36428)
0929 16:26:21.138879 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:21.138895 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:21.139029 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:21.139372 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:21.139375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:21.139382 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:26:21.139500 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:21.139506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:21.140372 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:21.140376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:21.141400 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:21.141411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:21.141847 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:21.141872 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:21.141961 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:21.141989 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:26:21.143879 (+ 1890us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40907/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 16:26:21.145830 (+ 1951us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43091: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40907/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":1675,"thread_start_us":115,"threads_started":1}
I20260929 16:26:21.146651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 5738
E20260929 16:26:21.155114 5835 webserver.cc:533] Webserver: sslize failed:
E20260929 16:26:21.155198 5835 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T16:26:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (324 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T16:26:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:26:21Z Disabled control of system clock
I20260929 16:26:21.194437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41395
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41395
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699152396020-392-0/testchainca.cert with env {}
W20260929 16:26:21.306434 5839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:21.306721 5839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:21.306762 5839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:21.310992 5839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:26:21.311064 5839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:21.311080 5839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:21.311101 5839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:26:21.311117 5839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:26:21.311134 5839 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:26:21.311147 5839 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 16:26:21.316754 5839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699152396020-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:21.317862 5839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:21.318872 5839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:26:21.327910 5839 server_base.cc:1034] running on GCE node
W20260929 16:26:21.327836 5845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:21.327857 5844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:21.327821 5847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:21.328759 5839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:21.329962 5839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:21.331171 5839 hybrid_clock.cc:648] HybridClock initialized: now 1790699181331137 us; error 52 us; skew 500 ppm
I20260929 16:26:21.333117 5839 webserver.cc:460] Webserver started at http://127.0.98.62:38175/ using document root <none> and password file <none>
I20260929 16:26:21.333841 5839 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:21.333906 5839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:21.334069 5839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:21.336115 5839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "ba61962ad2394b389d88e1867ae12ce7"
format_stamp: "Formatted at 2026-09-29 16:26:21 on dist-test-slave-vq9v"
server_key: "c6bf882ef07554044b5e2c4fb189b43e"
server_key_iv: "398e4cc75c6660e6b58f6ce247301602"
server_key_version: "encryptionkey@0"
I20260929 16:26:21.336709 5839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "ba61962ad2394b389d88e1867ae12ce7"
format_stamp: "Formatted at 2026-09-29 16:26:21 on dist-test-slave-vq9v"
server_key: "c6bf882ef07554044b5e2c4fb189b43e"
server_key_iv: "398e4cc75c6660e6b58f6ce247301602"
server_key_version: "encryptionkey@0"
I20260929 16:26:21.340649 5839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:26:21.343318 5853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:21.344709 5839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:26:21.344866 5839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "ba61962ad2394b389d88e1867ae12ce7"
format_stamp: "Formatted at 2026-09-29 16:26:21 on dist-test-slave-vq9v"
server_key: "c6bf882ef07554044b5e2c4fb189b43e"
server_key_iv: "398e4cc75c6660e6b58f6ce247301602"
server_key_version: "encryptionkey@0"
I20260929 16:26:21.344992 5839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:21.356621 5839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:21.357426 5839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:21.357633 5839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:21.364588 5839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41395
I20260929 16:26:21.364594 5905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41395 every 8 connection(s)
I20260929 16:26:21.365636 5839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:26:21.368769 5906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:21.370571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 5839
I20260929 16:26:21.370692 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:26:21.370975 392 external_mini_cluster.cc:1428] Setting key ec95a204da5f7e2e617406659ba39e14
I20260929 16:26:21.375546 5906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7: Bootstrap starting.
I20260929 16:26:21.377924 5906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:21.378815 5906 log.cc:824] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:21.380920 5909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:21.372408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.0.98.62:41395)
0929 16:26:21.373074 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:21.373099 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:21.373135 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:26:21.373875 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:21.373899 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:21.374376 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:21.374753 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:21.375196 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:21.376264 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:21.377180 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:21.377725 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:21.377904 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:21.378878 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:21.378884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:21.378897 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:21.379430 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:21.379501 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:21.379530 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:21.379641 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:21.379851 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 16:26:21.380199 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":126,"threads_started":1}
I20260929 16:26:21.381070 5906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7: No bootstrap required, opened a new log
I20260929 16:26:21.383755 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:26:21.384929 5906 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba61962ad2394b389d88e1867ae12ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } }
I20260929 16:26:21.385190 5906 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:21.385294 5906 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba61962ad2394b389d88e1867ae12ce7, State: Initialized, Role: FOLLOWER
I20260929 16:26:21.385766 5906 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba61962ad2394b389d88e1867ae12ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } }
I20260929 16:26:21.385922 5906 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:26:21.386011 5906 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:26:21.386116 5906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:21.387106 5906 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba61962ad2394b389d88e1867ae12ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } }
I20260929 16:26:21.387460 5906 leader_election.cc:302] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba61962ad2394b389d88e1867ae12ce7; no voters:
I20260929 16:26:21.387825 5906 leader_election.cc:288] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:26:21.387907 5911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:21.388224 5911 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7 [term 1 LEADER]: Becoming Leader. State: Replica: ba61962ad2394b389d88e1867ae12ce7, State: Running, Role: LEADER
I20260929 16:26:21.388558 5911 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba61962ad2394b389d88e1867ae12ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } }
I20260929 16:26:21.389127 5906 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:26:21.390352 5912 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba61962ad2394b389d88e1867ae12ce7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba61962ad2394b389d88e1867ae12ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } } }
I20260929 16:26:21.390532 5912 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:21.390887 5913 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba61962ad2394b389d88e1867ae12ce7. Latest consensus state: current_term: 1 leader_uuid: "ba61962ad2394b389d88e1867ae12ce7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba61962ad2394b389d88e1867ae12ce7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41395 } } }
I20260929 16:26:21.390985 5913 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7 [sys.catalog]: This master's current role is: LEADER
W20260929 16:26:21.393997 5926 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:26:21.394075 5926 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:26:21.394162 5921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:26:21.394855 5921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:26:21.400319 5921 catalog_manager.cc:1347] Generated new cluster ID: 4d3677b13a524a9397ae8a7bc333bb17
I20260929 16:26:21.400408 5921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:26:21.410277 5921 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:26:21.411408 5921 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:26:21.416366 5921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba61962ad2394b389d88e1867ae12ce7: Generated new TSK 0
I20260929 16:26:21.417460 5921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:26:21.516215 5909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:21.432927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48350 (local address 127.0.98.62:41395)
0929 16:26:21.433054 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:21.433058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:21.433256 (+ 198us) server_negotiation.cc:407] Connection header received
0929 16:26:21.433490 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:21.433494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:21.433559 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:21.433695 (+ 136us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:26:21.433945 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:21.434798 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:21.435695 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:21.436055 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:21.436158 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:21.436503 (+ 345us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:26:21.515693 (+ 79190us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 16:26:21.516044 (+ 351us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48350: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 16:26:21.516261 5935 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:21.432829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41395 (local address 127.0.0.1:48350)
0929 16:26:21.433103 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:21.433126 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:21.433328 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:21.433712 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:21.433715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:21.433723 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:26:21.433832 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:21.433838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:21.434927 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:21.434930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:21.435578 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:21.435586 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:21.436192 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:21.436214 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:21.436287 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:21.436311 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:26:21.515938 (+ 79627us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 16:26:21.516121 (+ 183us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41395: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":114,"threads_started":1}
W20260929 16:26:21.516309 5909 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48350: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260929 16:26:21.517053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 5839
2026-09-29T16:26:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (350 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (127 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (134 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:26:21 dist-test-slave-vq9v krb5kdc[5947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:26:21 dist-test-slave-vq9v krb5kdc[5947](info): set up 2 sockets
Sep 29 16:26:21 dist-test-slave-vq9v krb5kdc[5947](info): commencing operation
krb5kdc: starting...
W20260929 16:26:22.837848 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:26:22 dist-test-slave-vq9v krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:26:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:26:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:26:22.975864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39467
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:23.090790 5960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:23.091151 5960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:23.091229 5960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:23.095599 5960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:26:23.095728 5960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:23.095765 5960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:23.095806 5960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:26:23.095839 5960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:26:23.101531 5960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:23.102766 5960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:23.103791 5960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:23.113256 5965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:23.113324 5960 server_base.cc:1034] running on GCE node
W20260929 16:26:23.113539 5968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:23.113659 5966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:23.114326 5960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:23.115720 5960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:23.116989 5960 hybrid_clock.cc:648] HybridClock initialized: now 1790699183116968 us; error 39 us; skew 500 ppm
Sep 29 16:26:23 dist-test-slave-vq9v krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:23.120589 5960 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:26:23.122279 5960 webserver.cc:460] Webserver started at http://127.0.98.62:39463/ using document root <none> and password file <none>
I20260929 16:26:23.123070 5960 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:23.123199 5960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:23.123471 5960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:23.126112 5960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "4dc888d76dbb48bb8080032ab3ce9a9a"
format_stamp: "Formatted at 2026-09-29 16:26:23 on dist-test-slave-vq9v"
server_key: "a15509c1f513cf994991b6409ecacedb"
server_key_iv: "8a58e410d498afa7398397bbed36b561"
server_key_version: "encryptionkey@0"
I20260929 16:26:23.127003 5960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "4dc888d76dbb48bb8080032ab3ce9a9a"
format_stamp: "Formatted at 2026-09-29 16:26:23 on dist-test-slave-vq9v"
server_key: "a15509c1f513cf994991b6409ecacedb"
server_key_iv: "8a58e410d498afa7398397bbed36b561"
server_key_version: "encryptionkey@0"
I20260929 16:26:23.131753 5960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:26:23.134778 5975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:23.136359 5960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:26:23.136556 5960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "4dc888d76dbb48bb8080032ab3ce9a9a"
format_stamp: "Formatted at 2026-09-29 16:26:23 on dist-test-slave-vq9v"
server_key: "a15509c1f513cf994991b6409ecacedb"
server_key_iv: "8a58e410d498afa7398397bbed36b561"
server_key_version: "encryptionkey@0"
I20260929 16:26:23.136713 5960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:23.195679 5960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:23.199406 5960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:23.199656 5960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:23.207823 5960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39467
I20260929 16:26:23.208112 6027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39467 every 8 connection(s)
I20260929 16:26:23.208936 5960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:26:23.211925 6028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:23.213886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 5960
I20260929 16:26:23.214025 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:26:23.214309 392 external_mini_cluster.cc:1428] Setting key 8b7f23ebdf39e5b363bb9c6ab4e0e4f1
I20260929 16:26:23.219501 6028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a: Bootstrap starting.
I20260929 16:26:23.222407 6028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:23.223270 6028 log.cc:824] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:26:23 dist-test-slave-vq9v krb5kdc[5947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:23.225941 6028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a: No bootstrap required, opened a new log
I20260929 16:26:23.231482 6028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc888d76dbb48bb8080032ab3ce9a9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39467 } }
I20260929 16:26:23.231820 6028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:23.231876 6028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc888d76dbb48bb8080032ab3ce9a9a, State: Initialized, Role: FOLLOWER
I20260929 16:26:23.232154 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:23.215893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34516 (local address 127.0.98.62:39467)
0929 16:26:23.216359 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:23.216379 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:23.216422 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:26:23.217363 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:23.217391 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:23.217899 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:23.218431 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:23.219199 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:23.220596 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:23.221687 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:23.222430 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:23.222605 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:23.225834 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:23.225841 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:23.225860 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:23.225894 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:23.228202 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:23.228800 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:23.228809 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:23.228818 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:23.229036 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:23.229479 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:23.229483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:23.229485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:23.230107 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:23.230348 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:23.230667 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 16:26:23.231023 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":147,"threads_started":1}
I20260929 16:26:23.232398 6028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc888d76dbb48bb8080032ab3ce9a9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39467 } }
I20260929 16:26:23.232564 6028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:26:23.232650 6028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:26:23.232764 6028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:23.234144 6028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc888d76dbb48bb8080032ab3ce9a9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39467 } }
I20260929 16:26:23.234539 6028 leader_election.cc:302] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dc888d76dbb48bb8080032ab3ce9a9a; no voters:
I20260929 16:26:23.234941 6028 leader_election.cc:288] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:26:23.235064 6033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:23.235345 6033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a [term 1 LEADER]: Becoming Leader. State: Replica: 4dc888d76dbb48bb8080032ab3ce9a9a, State: Running, Role: LEADER
I20260929 16:26:23.235764 6033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc888d76dbb48bb8080032ab3ce9a9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39467 } }
I20260929 16:26:23.236387 6028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:26:23.237601 6034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dc888d76dbb48bb8080032ab3ce9a9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc888d76dbb48bb8080032ab3ce9a9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39467 } } }
I20260929 16:26:23.237802 6034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:23.238319 6035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dc888d76dbb48bb8080032ab3ce9a9a. Latest consensus state: current_term: 1 leader_uuid: "4dc888d76dbb48bb8080032ab3ce9a9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc888d76dbb48bb8080032ab3ce9a9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39467 } } }
I20260929 16:26:23.238503 6035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a [sys.catalog]: This master's current role is: LEADER
W20260929 16:26:23.241730 6048 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:26:23.241804 6048 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:26:23.241892 6042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:26:23.242656 6042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:26:23.248526 6042 catalog_manager.cc:1347] Generated new cluster ID: e0b438c48a7b4123a6256567ce31521b
I20260929 16:26:23.248628 6042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:26:23.269829 6042 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:26:23.270884 6042 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:26:23.291752 6042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dc888d76dbb48bb8080032ab3ce9a9a: Generated new TSK 0
I20260929 16:26:23.292613 6042 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 16:26:23.329416 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39467
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:23.446158 6054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:23.446431 6054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:23.446476 6054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:23.450577 6054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:23.450641 6054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:23.450702 6054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:26:23.456218 6054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:23.457304 6054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:23.458266 6054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:23.468436 6060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:23.468436 6059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:23.468652 6054 server_base.cc:1034] running on GCE node
W20260929 16:26:23.468449 6062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:23.469133 6054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:23.469789 6054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:23.470943 6054 hybrid_clock.cc:648] HybridClock initialized: now 1790699183470955 us; error 96 us; skew 500 ppm
Sep 29 16:26:23 dist-test-slave-vq9v krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:23.473860 6054 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:26:23.474740 6054 webserver.cc:460] Webserver started at http://127.0.98.1:36007/ using document root <none> and password file <none>
I20260929 16:26:23.475279 6054 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:23.475343 6054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:23.475512 6054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:23.477615 6054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "fd44e1dfb442432496ebc860c2ed3caf"
format_stamp: "Formatted at 2026-09-29 16:26:23 on dist-test-slave-vq9v"
server_key: "0877456dc17172749a903382c37c6905"
server_key_iv: "d678871c8f293498cb2347d2a09cb153"
server_key_version: "encryptionkey@0"
I20260929 16:26:23.478214 6054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fd44e1dfb442432496ebc860c2ed3caf"
format_stamp: "Formatted at 2026-09-29 16:26:23 on dist-test-slave-vq9v"
server_key: "0877456dc17172749a903382c37c6905"
server_key_iv: "d678871c8f293498cb2347d2a09cb153"
server_key_version: "encryptionkey@0"
I20260929 16:26:23.481976 6054 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:26:23.484651 6069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:23.486204 6054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:23.486318 6054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "fd44e1dfb442432496ebc860c2ed3caf"
format_stamp: "Formatted at 2026-09-29 16:26:23 on dist-test-slave-vq9v"
server_key: "0877456dc17172749a903382c37c6905"
server_key_iv: "d678871c8f293498cb2347d2a09cb153"
server_key_version: "encryptionkey@0"
I20260929 16:26:23.486418 6054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:23.501699 6054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:23.505159 6054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:23.505414 6054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:23.506028 6054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:23.506872 6054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:23.506945 6054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:23.507018 6054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:23.507061 6054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:23.516273 6054 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43247
I20260929 16:26:23.516305 6182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43247 every 8 connection(s)
I20260929 16:26:23.517175 6054 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:26:23.526228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 6054
I20260929 16:26:23.526386 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:26:23.526763 392 external_mini_cluster.cc:1428] Setting key 225d6f47eb5b585eb0ba19a8e956432f
Sep 29 16:26:23 dist-test-slave-vq9v krb5kdc[5947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:23.533138 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:23.519219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51517 (local address 127.0.98.62:39467)
0929 16:26:23.519381 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:23.519387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:23.520352 (+ 965us) server_negotiation.cc:407] Connection header received
0929 16:26:23.521796 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:23.521805 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:23.521861 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:23.521945 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:23.523274 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:23.523907 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:23.525127 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:23.525600 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:23.525705 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:23.528897 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:23.528903 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:23.528909 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:23.528944 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:23.530834 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:23.531531 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:23.531536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:23.531538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:23.531630 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:23.532009 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:23.532014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:23.532016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:23.532208 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:23.532318 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:23.532834 (+ 516us) server_negotiation.cc:299] Negotiation successful
0929 16:26:23.532970 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:26:23.534085 6185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:23.519694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39467 (local address 127.0.98.1:51517)
0929 16:26:23.520184 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:23.520228 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:23.521542 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:23.522118 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:23.522125 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:23.522530 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:23.523095 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:23.523112 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:23.524057 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:23.524060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:23.524988 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:23.524998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:23.525801 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:23.525805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:23.526138 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:23.526885 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:23.526909 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:23.528699 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:23.530970 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:23.530978 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:23.530997 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:23.531408 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:23.531729 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:23.531734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:23.531736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:23.531896 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:23.532317 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:23.532325 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:23.532584 (+ 259us) client_negotiation.cc:769] Sending connection context
0929 16:26:23.532784 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 16:26:23.533137 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":113,"threads_started":1}
I20260929 16:26:23.535451 6183 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39467
I20260929 16:26:23.535748 6183 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:23.536376 6183 heartbeater.cc:511] Master 127.0.98.62:39467 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 16:26:23.538193 5992 ts_manager.cc:194] Registered new tserver with Master: fd44e1dfb442432496ebc860c2ed3caf (127.0.98.1:43247)
I20260929 16:26:23.539426 5992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 16:26:23.557808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39467
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:23.670250 6188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:23.670560 6188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:23.670660 6188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:23.674763 6188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:23.674849 6188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:23.674970 6188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:26:23.680301 6188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:23.681475 6188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:23.682502 6188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:26:23.692641 6188 server_base.cc:1034] running on GCE node
W20260929 16:26:23.692587 6193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:23.692603 6194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:23.692603 6196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:23.693367 6188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:23.694037 6188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:23.695230 6188 hybrid_clock.cc:648] HybridClock initialized: now 1790699183695191 us; error 52 us; skew 500 ppm
Sep 29 16:26:23 dist-test-slave-vq9v krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:23.698194 6188 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:26:23.699059 6188 webserver.cc:460] Webserver started at http://127.0.98.2:35393/ using document root <none> and password file <none>
I20260929 16:26:23.699640 6188 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:23.699702 6188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:23.699914 6188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:23.702076 6188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "d62695b644b04f3a84f8af9274c00b88"
format_stamp: "Formatted at 2026-09-29 16:26:23 on dist-test-slave-vq9v"
server_key: "8b27ef7c1d5e2c1c1c2b816a8fd81710"
server_key_iv: "a70eadb1c2fb5f144f32b15bfaeba988"
server_key_version: "encryptionkey@0"
I20260929 16:26:23.702675 6188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d62695b644b04f3a84f8af9274c00b88"
format_stamp: "Formatted at 2026-09-29 16:26:23 on dist-test-slave-vq9v"
server_key: "8b27ef7c1d5e2c1c1c2b816a8fd81710"
server_key_iv: "a70eadb1c2fb5f144f32b15bfaeba988"
server_key_version: "encryptionkey@0"
I20260929 16:26:23.706669 6188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:26:23.709270 6203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:23.710701 6188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:23.710846 6188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "d62695b644b04f3a84f8af9274c00b88"
format_stamp: "Formatted at 2026-09-29 16:26:23 on dist-test-slave-vq9v"
server_key: "8b27ef7c1d5e2c1c1c2b816a8fd81710"
server_key_iv: "a70eadb1c2fb5f144f32b15bfaeba988"
server_key_version: "encryptionkey@0"
I20260929 16:26:23.710970 6188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:23.730782 6188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:23.734774 6188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:23.735028 6188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:23.735705 6188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:23.736589 6188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:23.736667 6188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:23.736739 6188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:23.736785 6188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:23.746995 6188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33861
I20260929 16:26:23.747066 6316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33861 every 8 connection(s)
I20260929 16:26:23.747987 6188 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:26:23.754766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 6188
I20260929 16:26:23.754894 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:26:23.755194 392 external_mini_cluster.cc:1428] Setting key a10dc556377406363601ab40a5f23d3a
Sep 29 16:26:23 dist-test-slave-vq9v krb5kdc[5947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 16:26:23.767199 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:23.749994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56355 (local address 127.0.98.62:39467)
0929 16:26:23.750160 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:23.750164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:23.750996 (+ 832us) server_negotiation.cc:407] Connection header received
0929 16:26:23.752465 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:23.752468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:23.752519 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:23.752639 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:23.753929 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:23.754540 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:23.756127 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:23.756377 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:23.756500 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:23.763213 (+ 6713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:23.763216 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:23.763218 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:23.763248 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:23.764862 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:23.765544 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:23.765550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:23.765551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:23.765634 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:23.766041 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:23.766044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:23.766046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:23.766247 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:23.766357 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:23.766897 (+ 540us) server_negotiation.cc:299] Negotiation successful
0929 16:26:23.767025 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 16:26:23.768168 6319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:23.750389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39467 (local address 127.0.98.2:56355)
0929 16:26:23.750839 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:23.750883 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:23.752226 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:23.752773 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:23.752780 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:23.753269 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:23.753782 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:23.753796 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:23.754694 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:23.754700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:23.755988 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:23.755999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:23.756554 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:23.756558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:23.756743 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:23.761038 (+ 4295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:23.761059 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:23.762937 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:23.764993 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:23.764999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:23.765015 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:23.765379 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:23.765770 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:23.765774 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:23.765776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:23.765927 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:23.766354 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:23.766362 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:23.766634 (+ 272us) client_negotiation.cc:769] Sending connection context
0929 16:26:23.766831 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 16:26:23.767191 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":108,"threads_started":1}
I20260929 16:26:23.769446 6317 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39467
I20260929 16:26:23.769738 6317 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:23.770236 6317 heartbeater.cc:511] Master 127.0.98.62:39467 requested a full tablet report, sending...
I20260929 16:26:23.771292 5992 ts_manager.cc:194] Registered new tserver with Master: d62695b644b04f3a84f8af9274c00b88 (127.0.98.2:33861)
I20260929 16:26:23.772204 5992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 16:26:23.782394 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39467
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:23.893761 6322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:23.894052 6322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:23.894097 6322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:23.898427 6322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:23.898499 6322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:23.898564 6322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:26:23.904167 6322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:23.905370 6322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:23.906495 6322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:23.916960 6328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:23.916975 6330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:23.916991 6327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:23.917331 6322 server_base.cc:1034] running on GCE node
I20260929 16:26:23.917940 6322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:23.918715 6322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:23.919935 6322 hybrid_clock.cc:648] HybridClock initialized: now 1790699183919895 us; error 53 us; skew 500 ppm
Sep 29 16:26:23 dist-test-slave-vq9v krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:23.923225 6322 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:26:23.924275 6322 webserver.cc:460] Webserver started at http://127.0.98.3:33639/ using document root <none> and password file <none>
I20260929 16:26:23.924892 6322 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:23.924994 6322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:23.925310 6322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:23.927599 6322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "d1fda514eaeb416d82ffdc9fe81d68a4"
format_stamp: "Formatted at 2026-09-29 16:26:23 on dist-test-slave-vq9v"
server_key: "2d8e5e92c4c0c04bec793d4ae1311790"
server_key_iv: "5e0035fb9d4ea95eab5c5ce725521c96"
server_key_version: "encryptionkey@0"
I20260929 16:26:23.928246 6322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d1fda514eaeb416d82ffdc9fe81d68a4"
format_stamp: "Formatted at 2026-09-29 16:26:23 on dist-test-slave-vq9v"
server_key: "2d8e5e92c4c0c04bec793d4ae1311790"
server_key_iv: "5e0035fb9d4ea95eab5c5ce725521c96"
server_key_version: "encryptionkey@0"
I20260929 16:26:23.932425 6322 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:26:23.935169 6337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:23.936503 6322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:26:23.936698 6322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "d1fda514eaeb416d82ffdc9fe81d68a4"
format_stamp: "Formatted at 2026-09-29 16:26:23 on dist-test-slave-vq9v"
server_key: "2d8e5e92c4c0c04bec793d4ae1311790"
server_key_iv: "5e0035fb9d4ea95eab5c5ce725521c96"
server_key_version: "encryptionkey@0"
I20260929 16:26:23.936865 6322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:23.948024 6322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:23.951692 6322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:23.951930 6322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:23.952579 6322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:23.953521 6322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:23.953594 6322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:23.953665 6322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:23.953722 6322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:23.963346 6322 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46363
I20260929 16:26:23.963387 6450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46363 every 8 connection(s)
I20260929 16:26:23.964298 6322 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:26:23.969076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 6322
I20260929 16:26:23.969272 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:26:23.969542 392 external_mini_cluster.cc:1428] Setting key 07a474b8eeeaea61c6531760cb1b3dba
Sep 29 16:26:23 dist-test-slave-vq9v krb5kdc[5947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:23.979149 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:23.966342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48623 (local address 127.0.98.62:39467)
0929 16:26:23.966523 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:23.966526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:23.967322 (+ 796us) server_negotiation.cc:407] Connection header received
0929 16:26:23.968751 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:23.968755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:23.968812 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:23.968901 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:23.970341 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:23.970979 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:23.972291 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:23.972524 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:23.972640 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:23.975224 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:23.975228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:23.975231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:23.975266 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:23.976841 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:23.977546 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:23.977550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:23.977551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:23.977645 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:23.978019 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:23.978022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:23.978024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:23.978220 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:23.978323 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:23.978854 (+ 531us) server_negotiation.cc:299] Negotiation successful
0929 16:26:23.978983 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:26:23.980001 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:23.966676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39467 (local address 127.0.98.3:48623)
0929 16:26:23.967153 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:23.967200 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:23.968531 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:23.969092 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:23.969099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:23.969576 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:23.970191 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:23.970210 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:23.971110 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:23.971113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:23.972158 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:23.972168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:23.972691 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:23.972696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:23.972881 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:23.973531 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:23.973548 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:23.975034 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:23.977066 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:23.977072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:23.977095 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:23.977436 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:23.977749 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:23.977752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:23.977754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:23.977914 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:23.978333 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:23.978342 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:23.978608 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 16:26:23.978802 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 16:26:23.979164 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":105,"threads_started":1}
I20260929 16:26:23.981345 6451 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39467
I20260929 16:26:23.981614 6451 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:23.982177 6451 heartbeater.cc:511] Master 127.0.98.62:39467 requested a full tablet report, sending...
I20260929 16:26:23.983268 5992 ts_manager.cc:194] Registered new tserver with Master: d1fda514eaeb416d82ffdc9fe81d68a4 (127.0.98.3:46363)
I20260929 16:26:23.984000 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:26:23.984140 5992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/test-admin.keytab.
Sep 29 16:26:23 dist-test-slave-vq9v krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:24.003214 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 16:26:24.006238 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:24.004627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52094 (local address 127.0.98.1:43247)
0929 16:26:24.004974 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:24.004989 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:24.005024 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:26:24.005076 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:24.005081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:24.005360 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:24.005491 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:24.006024 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52094: BlockingRecv error: recv got EOF from 127.0.0.1:52094 (error 108)
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":96,"threads_started":1}
W20260929 16:26:24.006537 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/krb5cc)
Sep 29 16:26:24 dist-test-slave-vq9v krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:24.007800 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 16:26:24 dist-test-slave-vq9v krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:24.008936 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 16:26:24 dist-test-slave-vq9v krb5kdc[5947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 16:26:24.020392 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:24.010127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52110 (local address 127.0.98.1:43247)
0929 16:26:24.010304 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:24.010308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:24.010323 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:26:24.010440 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:24.010443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:24.010485 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:24.010556 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:24.011163 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:24.011979 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:24.012930 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:24.013432 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:24.013530 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:24.016257 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:24.016267 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:24.016281 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:24.016308 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:24.018145 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:24.018714 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:24.018721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:24.018722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:24.018825 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:24.019227 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:24.019230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:24.019232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:24.019503 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:24.019677 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:24.019950 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 16:26:24.020241 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260929 16:26:24.025615 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:24.024246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52118 (local address 127.0.98.1:43247)
0929 16:26:24.024462 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:24.024467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:24.024585 (+ 118us) server_negotiation.cc:407] Connection header received
0929 16:26:24.024760 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:24.024763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:24.024812 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:24.024902 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:24.025485 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52118: BlockingRecv error: recv got EOF from 127.0.0.1:52118 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260929 16:26:24.025887 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 16:26:24 dist-test-slave-vq9v krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:24.027034 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 16:26:24 dist-test-slave-vq9v krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:24.027956 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 16:26:24 dist-test-slave-vq9v krb5kdc[5947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 16:26:24.038491 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:24.028947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52124 (local address 127.0.98.1:43247)
0929 16:26:24.029115 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:24.029120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:24.029346 (+ 226us) server_negotiation.cc:407] Connection header received
0929 16:26:24.029490 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:24.029493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:24.029534 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:24.029627 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:24.030189 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:24.030750 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:24.031598 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:24.031863 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:24.031983 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:24.034707 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:24.034711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:24.034713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:24.034747 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:24.036511 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:24.037099 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:24.037144 (+ 45us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:24.037146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:24.037290 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:24.037591 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:24.037595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:24.037597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:24.037808 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:24.037949 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:24.038201 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 16:26:24.038325 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 16:26:24.042081 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:24.040570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52138 (local address 127.0.98.1:43247)
0929 16:26:24.040717 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:24.040723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:24.041068 (+ 345us) server_negotiation.cc:407] Connection header received
0929 16:26:24.041329 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:24.041331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:24.041377 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:24.041452 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:24.041955 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52138: BlockingRecv error: recv got EOF from 127.0.0.1:52138 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 16:26:24.042158 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790699152396020-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 16:26:24 dist-test-slave-vq9v krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:24.043258 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 16:26:24 dist-test-slave-vq9v krb5kdc[5947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:24.044183 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 16:26:24 dist-test-slave-vq9v krb5kdc[5947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 16:26:24.054018 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:24.045032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.0.98.1:43247)
0929 16:26:24.045180 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:24.045186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:24.045337 (+ 151us) server_negotiation.cc:407] Connection header received
0929 16:26:24.045416 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:24.045419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:24.045476 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:24.045558 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:24.046016 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:24.046550 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:24.047366 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:24.047621 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:24.047726 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:24.050332 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:24.050336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:24.050338 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:24.050369 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:24.052064 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:24.052546 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:24.052549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:24.052550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:24.052646 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:24.052917 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:24.052920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:24.052922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:24.053161 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:24.053385 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:24.053654 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 16:26:24.053815 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:26:24.055359 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 6054
I20260929 16:26:24.061093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 6188
I20260929 16:26:24.066753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 6322
I20260929 16:26:24.072715 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 5960
2026-09-29T16:26:24Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2297 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:26:24 dist-test-slave-vq9v krb5kdc[6471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:26:24 dist-test-slave-vq9v krb5kdc[6471](info): set up 2 sockets
Sep 29 16:26:24 dist-test-slave-vq9v krb5kdc[6471](info): commencing operation
krb5kdc: starting...
W20260929 16:26:25.117690 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:26:25 dist-test-slave-vq9v krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:26:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:26:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260929 16:26:25.245476 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44555
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:25.356374 6484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:25.356720 6484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:25.356807 6484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:25.360931 6484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:26:25.361048 6484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:25.361086 6484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:25.361135 6484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:26:25.361174 6484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:26:25.366719 6484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:25.367875 6484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:25.368927 6484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:25.377776 6492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:25.377776 6490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:25.377899 6489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:25.377825 6484 server_base.cc:1034] running on GCE node
I20260929 16:26:25.378628 6484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:25.379788 6484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:25.381008 6484 hybrid_clock.cc:648] HybridClock initialized: now 1790699185380979 us; error 44 us; skew 500 ppm
Sep 29 16:26:25 dist-test-slave-vq9v krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699185, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:25.384172 6484 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260929 16:26:25.385169 6484 webserver.cc:460] Webserver started at http://127.0.98.62:35913/ using document root <none> and password file <none>
I20260929 16:26:25.385855 6484 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:25.385960 6484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:25.386215 6484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:25.388366 6484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "6d732cddd2a8490ea5a12bdcc1798761"
format_stamp: "Formatted at 2026-09-29 16:26:25 on dist-test-slave-vq9v"
server_key: "40ea69f8650fecd6ee5df7dd866c2b21"
server_key_iv: "9f586c3fa2c7a993f11968206704ac75"
server_key_version: "encryptionkey@0"
I20260929 16:26:25.388981 6484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "6d732cddd2a8490ea5a12bdcc1798761"
format_stamp: "Formatted at 2026-09-29 16:26:25 on dist-test-slave-vq9v"
server_key: "40ea69f8650fecd6ee5df7dd866c2b21"
server_key_iv: "9f586c3fa2c7a993f11968206704ac75"
server_key_version: "encryptionkey@0"
I20260929 16:26:25.393121 6484 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260929 16:26:25.395957 6499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:25.397375 6484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:26:25.397519 6484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "6d732cddd2a8490ea5a12bdcc1798761"
format_stamp: "Formatted at 2026-09-29 16:26:25 on dist-test-slave-vq9v"
server_key: "40ea69f8650fecd6ee5df7dd866c2b21"
server_key_iv: "9f586c3fa2c7a993f11968206704ac75"
server_key_version: "encryptionkey@0"
I20260929 16:26:25.397644 6484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:25.418623 6484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:25.422324 6484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:25.422550 6484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:25.429968 6484 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44555
I20260929 16:26:25.430373 6551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44555 every 8 connection(s)
I20260929 16:26:25.431022 6484 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:26:25.432477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 6484
I20260929 16:26:25.432628 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:26:25.432911 392 external_mini_cluster.cc:1428] Setting key 6ac043d24f25c6fcc477ddf7ac46010b
I20260929 16:26:25.434322 6552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:25.441085 6552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761: Bootstrap starting.
Sep 29 16:26:25 dist-test-slave-vq9v krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 16:26:25.443761 6552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:25.444761 6552 log.cc:824] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:25.447135 6552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761: No bootstrap required, opened a new log
I20260929 16:26:25.448459 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:25.434483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42466 (local address 127.0.98.62:44555)
0929 16:26:25.434903 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:25.434921 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:25.434963 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:26:25.435839 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:25.435861 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:25.436285 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:25.436658 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:25.437449 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:25.438942 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:25.439817 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:25.440528 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:25.440686 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:25.443444 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:25.443450 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:25.443467 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:25.443500 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:25.445644 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:25.446072 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:25.446078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:25.446084 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:25.446227 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:25.446480 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:25.446483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:25.446485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:25.446978 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:25.447172 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:25.447453 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 16:26:25.447749 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":118,"threads_started":1}
I20260929 16:26:25.450884 6552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } }
I20260929 16:26:25.451114 6552 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:25.451184 6552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d732cddd2a8490ea5a12bdcc1798761, State: Initialized, Role: FOLLOWER
I20260929 16:26:25.451586 6552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } }
I20260929 16:26:25.451716 6552 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:26:25.451786 6552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:26:25.451903 6552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:25.452867 6552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } }
I20260929 16:26:25.453244 6552 leader_election.cc:302] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d732cddd2a8490ea5a12bdcc1798761; no voters:
I20260929 16:26:25.453601 6552 leader_election.cc:288] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:26:25.453709 6557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:25.453884 6557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [term 1 LEADER]: Becoming Leader. State: Replica: 6d732cddd2a8490ea5a12bdcc1798761, State: Running, Role: LEADER
I20260929 16:26:25.454185 6557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } }
I20260929 16:26:25.454914 6552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:26:25.456029 6559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d732cddd2a8490ea5a12bdcc1798761. Latest consensus state: current_term: 1 leader_uuid: "6d732cddd2a8490ea5a12bdcc1798761" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } }
I20260929 16:26:25.456039 6558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d732cddd2a8490ea5a12bdcc1798761" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } }
I20260929 16:26:25.456290 6559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:25.456323 6558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [sys.catalog]: This master's current role is: LEADER
W20260929 16:26:25.460282 6573 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:26:25.460376 6573 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:26:25.460517 6566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:26:25.461355 6566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:26:25.467273 6566 catalog_manager.cc:1347] Generated new cluster ID: d3fbbee867e94a12b43864bbe215c0dc
I20260929 16:26:25.467370 6566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:26:25.476743 6566 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:26:25.477821 6566 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:26:25.484129 6566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761: Generated new TSK 0
I20260929 16:26:25.484826 6566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260929 16:26:25.520767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44555
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:25.654165 6578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:25.654443 6578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:25.654485 6578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:25.658668 6578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:25.658757 6578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:25.658854 6578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:26:25.664181 6578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:25.665354 6578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:25.666422 6578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:25.676657 6586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:25.676641 6584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:25.676641 6583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:25.676878 6578 server_base.cc:1034] running on GCE node
I20260929 16:26:25.677515 6578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:25.678226 6578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:25.679450 6578 hybrid_clock.cc:648] HybridClock initialized: now 1790699185679433 us; error 33 us; skew 500 ppm
Sep 29 16:26:25 dist-test-slave-vq9v krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699185, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:25.682497 6578 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 16:26:25.683413 6578 webserver.cc:460] Webserver started at http://127.0.98.1:43521/ using document root <none> and password file <none>
I20260929 16:26:25.684017 6578 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:25.684108 6578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:25.684340 6578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:25.686451 6578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "13adcbd1e74248ab871d13068c0f23c9"
format_stamp: "Formatted at 2026-09-29 16:26:25 on dist-test-slave-vq9v"
server_key: "c368fc09d1ba5da71283a3bb37d57b02"
server_key_iv: "2cd8580d4e56f07867c26b816955c94c"
server_key_version: "encryptionkey@0"
I20260929 16:26:25.687021 6578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "13adcbd1e74248ab871d13068c0f23c9"
format_stamp: "Formatted at 2026-09-29 16:26:25 on dist-test-slave-vq9v"
server_key: "c368fc09d1ba5da71283a3bb37d57b02"
server_key_iv: "2cd8580d4e56f07867c26b816955c94c"
server_key_version: "encryptionkey@0"
I20260929 16:26:25.690897 6578 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:26:25.693459 6593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:25.694725 6578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:26:25.694852 6578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "13adcbd1e74248ab871d13068c0f23c9"
format_stamp: "Formatted at 2026-09-29 16:26:25 on dist-test-slave-vq9v"
server_key: "c368fc09d1ba5da71283a3bb37d57b02"
server_key_iv: "2cd8580d4e56f07867c26b816955c94c"
server_key_version: "encryptionkey@0"
I20260929 16:26:25.694960 6578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:25.728034 6578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:25.731910 6578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:25.732161 6578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:25.732882 6578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:25.733831 6578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:25.733903 6578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:25.733975 6578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:25.734016 6578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:25.743957 6578 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41403
I20260929 16:26:25.743979 6706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41403 every 8 connection(s)
I20260929 16:26:25.744984 6578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:26:25.753157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 6578
I20260929 16:26:25.753351 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:26:25.753692 392 external_mini_cluster.cc:1428] Setting key e942d623fb90778d38a989911dff5128
Sep 29 16:26:25 dist-test-slave-vq9v krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699185, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 16:26:25.760581 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:25.747001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38575 (local address 127.0.98.62:44555)
0929 16:26:25.747152 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:25.747157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:25.748108 (+ 951us) server_negotiation.cc:407] Connection header received
0929 16:26:25.749659 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:25.749666 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:25.749738 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:25.749823 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:25.751277 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:25.751865 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:25.753426 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:25.753754 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:25.753904 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:25.756747 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:25.756754 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:25.756757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:25.756785 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:25.758360 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:25.758941 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:25.758949 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:25.758951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:25.759030 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:25.759399 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:25.759405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:25.759406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:25.759602 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:25.759722 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:25.760276 (+ 554us) server_negotiation.cc:299] Negotiation successful
0929 16:26:25.760415 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:26:25.761567 6709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:25.747346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.1:38575)
0929 16:26:25.747908 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:25.747962 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:25.749399 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:25.750041 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:25.750049 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:25.750489 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:25.751085 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:25.751102 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:25.752021 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:25.752025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:25.753262 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:25.753279 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:25.753956 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:25.753960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:25.754202 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:25.754879 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:25.754896 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:25.756542 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:25.758499 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:25.758506 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:25.758525 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:25.758818 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:25.759141 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:25.759144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:25.759146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:25.759301 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:25.759720 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:25.759729 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:25.760015 (+ 286us) client_negotiation.cc:769] Sending connection context
0929 16:26:25.760225 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 16:26:25.760582 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":120,"threads_started":1}
I20260929 16:26:25.763212 6707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44555
I20260929 16:26:25.763499 6707 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 16:26:25.764151 6707 heartbeater.cc:511] Master 127.0.98.62:44555 requested a full tablet report, sending...
I20260929 16:26:25.765957 6516 ts_manager.cc:194] Registered new tserver with Master: 13adcbd1e74248ab871d13068c0f23c9 (127.0.98.1:41403)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260929 16:26:25.767333 6516 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260929 16:26:25.785812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44555
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:25.897454 6712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:25.897778 6712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:25.897850 6712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:25.902315 6712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:25.902410 6712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:25.902522 6712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:26:25.907990 6712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:25.909269 6712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:25.910336 6712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:26:25.920398 6712 server_base.cc:1034] running on GCE node
W20260929 16:26:25.920446 6717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:25.920416 6718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:25.920434 6720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:25.921051 6712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:25.921850 6712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:25.923053 6712 hybrid_clock.cc:648] HybridClock initialized: now 1790699185923030 us; error 37 us; skew 500 ppm
Sep 29 16:26:25 dist-test-slave-vq9v krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699185, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:25.926270 6712 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 16:26:25.927271 6712 webserver.cc:460] Webserver started at http://127.0.98.2:45099/ using document root <none> and password file <none>
I20260929 16:26:25.927888 6712 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:25.927991 6712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:25.928241 6712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:25.930517 6712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "bcbdbf5a43df466cb16a8911c91e24a6"
format_stamp: "Formatted at 2026-09-29 16:26:25 on dist-test-slave-vq9v"
server_key: "58297428cc5bad00e91a09c94c3d32d3"
server_key_iv: "11295fe37359087bf82523a53aba285d"
server_key_version: "encryptionkey@0"
I20260929 16:26:25.931386 6712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bcbdbf5a43df466cb16a8911c91e24a6"
format_stamp: "Formatted at 2026-09-29 16:26:25 on dist-test-slave-vq9v"
server_key: "58297428cc5bad00e91a09c94c3d32d3"
server_key_iv: "11295fe37359087bf82523a53aba285d"
server_key_version: "encryptionkey@0"
I20260929 16:26:25.936024 6712 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:26:25.938891 6727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:25.940460 6712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:26:25.940649 6712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "bcbdbf5a43df466cb16a8911c91e24a6"
format_stamp: "Formatted at 2026-09-29 16:26:25 on dist-test-slave-vq9v"
server_key: "58297428cc5bad00e91a09c94c3d32d3"
server_key_iv: "11295fe37359087bf82523a53aba285d"
server_key_version: "encryptionkey@0"
I20260929 16:26:25.940835 6712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:25.976583 6712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:25.980141 6712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:25.980379 6712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:25.980989 6712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:25.981948 6712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:25.982020 6712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:25.982097 6712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:25.982149 6712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:25.991657 6712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43549
I20260929 16:26:25.991694 6840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43549 every 8 connection(s)
I20260929 16:26:25.992601 6712 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:26:25.993124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 6712
I20260929 16:26:25.993261 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:26:25.993621 392 external_mini_cluster.cc:1428] Setting key 72035e02e671872ac33023e3661718f9
Sep 29 16:26:25 dist-test-slave-vq9v krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699185, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260929 16:26:26.010237 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:25.994646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44495 (local address 127.0.98.62:44555)
0929 16:26:25.994793 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:25.994798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:25.995861 (+ 1063us) server_negotiation.cc:407] Connection header received
0929 16:26:25.997958 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:25.997962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:25.998021 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:25.998127 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:25.999723 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.000300 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.002289 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.002518 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.002603 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.005711 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:26.005717 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:26.005720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:26.005757 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:26.007434 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.008120 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.008123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.008124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.008240 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.008709 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.008714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.008718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.008994 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:26.009106 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:26.009950 (+ 844us) server_negotiation.cc:299] Negotiation successful
0929 16:26:26.010070 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:26:26.011412 6843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:25.995101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.2:44495)
0929 16:26:25.995652 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:25.995718 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:25.997719 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:25.998414 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:25.998427 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:25.998892 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:25.999539 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:25.999554 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.000437 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.000444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.002107 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.002135 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.002696 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.002705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.002984 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.003674 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:26.003699 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:26.005516 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:26.007585 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.007593 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.007609 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.007997 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.008359 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.008362 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.008364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.008525 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.009167 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:26.009176 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:26.009598 (+ 422us) client_negotiation.cc:769] Sending connection context
0929 16:26:26.009845 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 16:26:26.010313 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":115,"threads_started":1}
I20260929 16:26:26.013025 6841 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44555
I20260929 16:26:26.013432 6841 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:26.013983 6841 heartbeater.cc:511] Master 127.0.98.62:44555 requested a full tablet report, sending...
I20260929 16:26:26.015179 6516 ts_manager.cc:194] Registered new tserver with Master: bcbdbf5a43df466cb16a8911c91e24a6 (127.0.98.2:43549)
I20260929 16:26:26.016027 6516 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 16:26:26.029168 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44555
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:26.144091 6846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:26.144526 6846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:26.144613 6846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:26.148885 6846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:26.149004 6846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:26.149106 6846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:26:26.154676 6846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:26.155994 6846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:26.157158 6846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:26.167400 6854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:26.167563 6846 server_base.cc:1034] running on GCE node
W20260929 16:26:26.167444 6851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:26.167448 6852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:26.168308 6846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:26.169093 6846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:26.170305 6846 hybrid_clock.cc:648] HybridClock initialized: now 1790699186170269 us; error 55 us; skew 500 ppm
Sep 29 16:26:26 dist-test-slave-vq9v krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699186, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:26.173247 6846 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 16:26:26.174165 6846 webserver.cc:460] Webserver started at http://127.0.98.3:44219/ using document root <none> and password file <none>
I20260929 16:26:26.174690 6846 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:26.174749 6846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:26.174958 6846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:26.176976 6846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "4bcbbb5789854afbbea071a979eccfb7"
format_stamp: "Formatted at 2026-09-29 16:26:26 on dist-test-slave-vq9v"
server_key: "68f0b8011771afd7a384a7a6e0667374"
server_key_iv: "809f34e05188ab3f9d1369c70c07e04b"
server_key_version: "encryptionkey@0"
I20260929 16:26:26.177601 6846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4bcbbb5789854afbbea071a979eccfb7"
format_stamp: "Formatted at 2026-09-29 16:26:26 on dist-test-slave-vq9v"
server_key: "68f0b8011771afd7a384a7a6e0667374"
server_key_iv: "809f34e05188ab3f9d1369c70c07e04b"
server_key_version: "encryptionkey@0"
I20260929 16:26:26.181448 6846 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:26:26.183988 6861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:26.185556 6846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:26.185702 6846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "4bcbbb5789854afbbea071a979eccfb7"
format_stamp: "Formatted at 2026-09-29 16:26:26 on dist-test-slave-vq9v"
server_key: "68f0b8011771afd7a384a7a6e0667374"
server_key_iv: "809f34e05188ab3f9d1369c70c07e04b"
server_key_version: "encryptionkey@0"
I20260929 16:26:26.185837 6846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:26.199452 6846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:26.203104 6846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:26.203361 6846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:26.204316 6846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:26.205649 6846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:26.205708 6846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:26.205766 6846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:26.205799 6846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:26.217661 6846 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46539
I20260929 16:26:26.217782 6974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46539 every 8 connection(s)
I20260929 16:26:26.218631 6846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:26:26.226248 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 6846
I20260929 16:26:26.226349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:26:26.226644 392 external_mini_cluster.cc:1428] Setting key 42da922b3d5b85fd89ae8d8cca4c595e
Sep 29 16:26:26 dist-test-slave-vq9v krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699186, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 16:26:26.234359 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.220680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48979 (local address 127.0.98.62:44555)
0929 16:26:26.220823 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:26.220827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:26.221838 (+ 1011us) server_negotiation.cc:407] Connection header received
0929 16:26:26.223308 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:26.223311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:26.223370 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:26.223485 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:26.224688 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.225298 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.226542 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.226830 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.226952 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.230181 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:26.230185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:26.230188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:26.230218 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:26.231880 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.232564 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.232567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.232569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.232685 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.233082 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.233085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.233087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.233420 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:26.233530 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:26.234109 (+ 579us) server_negotiation.cc:299] Negotiation successful
0929 16:26:26.234236 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:26:26.235385 6977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.221099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.3:48979)
0929 16:26:26.221670 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:26.221719 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:26.223070 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:26.223665 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:26.223672 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:26.224044 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:26.224532 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.224545 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.225429 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.225438 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.226419 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.226428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.227015 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.227019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.227194 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.227994 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:26.228038 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:26.229945 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:26.232026 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.232032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.232047 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.232440 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.232783 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.232786 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.232788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.232957 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.233535 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:26.233542 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:26.233833 (+ 291us) client_negotiation.cc:769] Sending connection context
0929 16:26:26.234023 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 16:26:26.234333 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":121,"threads_started":1}
I20260929 16:26:26.236750 6975 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44555
I20260929 16:26:26.237044 6975 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:26.237654 6975 heartbeater.cc:511] Master 127.0.98.62:44555 requested a full tablet report, sending...
I20260929 16:26:26.238775 6516 ts_manager.cc:194] Registered new tserver with Master: 4bcbbb5789854afbbea071a979eccfb7 (127.0.98.3:46539)
I20260929 16:26:26.239423 6516 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48979
I20260929 16:26:26.240877 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260929 16:26:26.273838 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36225
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44555,127.0.98.61:36225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:26.387773 6980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:26.388099 6980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:26.388186 6980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:26.392510 6980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:26:26.392616 6980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:26.392652 6980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:26.392714 6980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:26:26.392767 6980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:26:26.398542 6980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44555,127.0.98.61:36225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
W20260929 16:26:26.398841 6980 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:44555,127.0.98.61:36225'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 16:26:26.399866 6980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:26.401057 6980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:26.410315 6986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:26.410254 6985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:26.410555 6980 server_base.cc:1034] running on GCE node
W20260929 16:26:26.410254 6988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:26.411252 6980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:26.412549 6980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:26.413753 6980 hybrid_clock.cc:648] HybridClock initialized: now 1790699186413706 us; error 66 us; skew 500 ppm
Sep 29 16:26:26 dist-test-slave-vq9v krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699186, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:26.416961 6980 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260929 16:26:26.418031 6980 webserver.cc:460] Webserver started at http://127.0.98.61:36999/ using document root <none> and password file <none>
I20260929 16:26:26.418618 6980 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:26.418685 6980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:26.418870 6980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:26.421373 6980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/data/instance:
uuid: "7bc65de05f0c4d39a89442538492b338"
format_stamp: "Formatted at 2026-09-29 16:26:26 on dist-test-slave-vq9v"
server_key: "b0e6f620ac76759735ca29eb535e4eca"
server_key_iv: "9c26e34e54490442f58c9b2d3ecadcbc"
server_key_version: "encryptionkey@0"
I20260929 16:26:26.422055 6980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/wal/instance:
uuid: "7bc65de05f0c4d39a89442538492b338"
format_stamp: "Formatted at 2026-09-29 16:26:26 on dist-test-slave-vq9v"
server_key: "b0e6f620ac76759735ca29eb535e4eca"
server_key_iv: "9c26e34e54490442f58c9b2d3ecadcbc"
server_key_version: "encryptionkey@0"
I20260929 16:26:26.426542 6980 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:26:26.430001 6995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:26.431833 6980 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:26:26.432015 6980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/wal
uuid: "7bc65de05f0c4d39a89442538492b338"
format_stamp: "Formatted at 2026-09-29 16:26:26 on dist-test-slave-vq9v"
server_key: "b0e6f620ac76759735ca29eb535e4eca"
server_key_iv: "9c26e34e54490442f58c9b2d3ecadcbc"
server_key_version: "encryptionkey@0"
I20260929 16:26:26.432153 6980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:26.448683 6980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:26.451983 6980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:26.452152 6980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:26.459947 6980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36225
I20260929 16:26:26.460299 7047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36225 every 8 connection(s)
I20260929 16:26:26.461011 6980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/data/info.pb
I20260929 16:26:26.463995 7048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:26.467818 7048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:26:26.470878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 6980
Sep 29 16:26:26 dist-test-slave-vq9v krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699186, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 16:26:26 dist-test-slave-vq9v krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 16:26:26.483079 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.469907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43188 (local address 127.0.98.62:44555)
0929 16:26:26.470045 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:26.470048 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:26.470961 (+ 913us) server_negotiation.cc:407] Connection header received
0929 16:26:26.472301 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:26.472307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:26.472366 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:26.472425 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:26.473931 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.474542 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.475743 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.475982 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.476123 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.478930 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:26.478933 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:26.478935 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:26.478961 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:26.480630 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.481292 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.481296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.481298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.481387 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.481786 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.481791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.481795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.482099 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:26.482185 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:26.482748 (+ 563us) server_negotiation.cc:299] Negotiation successful
0929 16:26:26.482891 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:26:26.484170 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.470279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.0.1:43188)
0929 16:26:26.470802 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:26.470849 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:26.471935 (+ 1086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:26.472684 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:26.472696 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:26.473150 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:26.473629 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.473644 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.474688 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.474691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.475576 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.475584 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.476151 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.476155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.476269 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.477028 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:26.477044 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:26.478736 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:26.480775 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.480784 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.480797 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.481156 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.481485 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.481487 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.481489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.481661 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.482199 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:26.482206 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:26.482496 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 16:26:26.482667 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 16:26:26.483044 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":106,"spinlock_wait_cycles":1792,"thread_start_us":144,"threads_started":1}
I20260929 16:26:26.484170 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.471813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59418 (local address 127.0.98.61:36225)
0929 16:26:26.472195 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:26.472206 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:26.472247 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:26:26.472438 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:26.472458 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:26.472684 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:26.472880 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:26.473728 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.474680 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.475576 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.476004 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.476187 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.479323 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:26.479329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:26.479342 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:26.479362 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:26.481101 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.481635 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.481641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.481643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.481720 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.481987 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.481989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.481990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.482495 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:26.482696 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:26.482936 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 16:26:26.483091 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":266,"spinlock_wait_cycles":1792,"thread_start_us":173,"threads_started":1}
I20260929 16:26:26.485729 7048 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:26:26 dist-test-slave-vq9v krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699186, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 16:26:26.494369 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.486210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36225 (local address 127.0.0.1:59434)
0929 16:26:26.486399 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:26.486414 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:26.486480 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:26.486753 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:26.486755 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:26.486897 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:26.486993 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.486997 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.487720 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.487723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.488262 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.488270 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.488793 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.488796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.488886 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.489517 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:26.489528 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:26.490981 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:26.492842 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.492845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.492846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.493049 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.493446 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.493448 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.493449 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.493525 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.493897 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:26.493900 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:26.493982 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 16:26:26.494070 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 16:26:26.494174 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260929 16:26:26.494395 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.486354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.0.98.61:36225)
0929 16:26:26.486476 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:26.486481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:26.486492 (+ 11us) server_negotiation.cc:407] Connection header received
0929 16:26:26.486574 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:26.486577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:26.486621 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:26.486727 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:26.487092 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.487619 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.488385 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.488656 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.488748 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.491178 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:26.491181 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:26.491184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:26.491206 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:26.492723 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.493193 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.493260 (+ 67us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.493262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.493357 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.493612 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.493615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.493617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.493806 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:26.493887 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:26.494159 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 16:26:26.494270 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 16:26:26.498021 7048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338: Bootstrap starting.
I20260929 16:26:26.500794 7048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:26.501837 7048 log.cc:824] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:26.503970 7048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338: No bootstrap required, opened a new log
I20260929 16:26:26.507124 7048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } }
I20260929 16:26:26.507383 7048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:26.507467 7048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bc65de05f0c4d39a89442538492b338, State: Initialized, Role: FOLLOWER
I20260929 16:26:26.507910 7048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } }
I20260929 16:26:26.508380 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } } }
I20260929 16:26:26.508505 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:26:26.509152 7048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 16:26:26.514081 7065 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:26:26.514173 7065 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:26:26.515944 6980 master_runner.cc:428] Detected that this master 7bc65de05f0c4d39a89442538492b338 is joining an existing cluster
I20260929 16:26:26.515995 6980 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:36225
I20260929 16:26:26.526705 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.518543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43204 (local address 127.0.98.62:44555)
0929 16:26:26.518768 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:26.518772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:26.518784 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:26:26.518826 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:26.518828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:26.518873 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:26.518951 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:26.519557 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.520346 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.521260 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.521503 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.521621 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.523049 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:26.523053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:26.523056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:26.523083 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:26.524832 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.525403 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.525407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.525409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.525500 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.525783 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.525786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.525788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.526033 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:26.526136 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:26.526412 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 16:26:26.526536 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260929 16:26:26.526762 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.518513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.0.98.61:36225)
0929 16:26:26.518663 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:26.518667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:26.518743 (+ 76us) server_negotiation.cc:407] Connection header received
0929 16:26:26.518834 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:26.518836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:26.518873 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:26.518951 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:26.519582 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.520346 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.521615 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.521945 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.522013 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.523268 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:26.523274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:26.523280 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:26.523307 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:26.525015 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.525537 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.525540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.525542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.525622 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.525895 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.525898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.525900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.526068 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:26.526175 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:26.526428 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 16:26:26.526568 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":84,"spinlock_wait_cycles":2304}
I20260929 16:26:26.527053 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.518136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.0.1:43204)
0929 16:26:26.518451 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:26.518469 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:26.518569 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:26.519073 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:26.519076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:26.519268 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:26.519432 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.519438 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.520473 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.520476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.521117 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.521124 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.521618 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.521621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.521719 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.522277 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:26.522290 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:26.522848 (+ 558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:26.524961 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.524964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.524967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.525294 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.525578 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.525584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.525588 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.525699 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.526139 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:26.526144 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:26.526229 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 16:26:26.526347 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 16:26:26.526904 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":74,"spinlock_wait_cycles":384,"thread_start_us":146,"threads_started":1}
I20260929 16:26:26.526775 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.518236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36225 (local address 127.0.0.1:59436)
0929 16:26:26.518648 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:26.518662 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:26.518765 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:26.518992 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:26.518995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:26.519255 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:26.519449 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.519466 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.520464 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.520467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.521494 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.521504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.522138 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.522153 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.522237 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.522712 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:26.522728 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:26.523124 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:26.525117 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.525123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.525125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.525438 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.525722 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.525728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.525733 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.525814 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.526215 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:26.526218 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:26.526280 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 16:26:26.526482 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 16:26:26.526620 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":83,"spinlock_wait_cycles":4864,"thread_start_us":100,"threads_started":1}
Sep 29 16:26:26 dist-test-slave-vq9v krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699185, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 16:26:26.542511 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.532381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59446 (local address 127.0.98.61:36225)
0929 16:26:26.532517 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:26.532522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:26.532882 (+ 360us) server_negotiation.cc:407] Connection header received
0929 16:26:26.533063 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:26.533066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:26.533110 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:26.533192 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:26.533988 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.534500 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.535739 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.535947 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.536028 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.538815 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:26.538818 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:26.538820 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:26.538840 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:26.540583 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.541132 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.541135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.541137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.541275 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.541579 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.541584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.541585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.541772 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:26.541878 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:26.542230 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 16:26:26.542348 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:26:26.542527 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.532337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36225 (local address 127.0.0.1:59446)
0929 16:26:26.532741 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:26.532794 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:26.532935 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:26.533256 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:26.533262 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:26.533709 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:26.533847 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.533855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.534608 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.534612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.535618 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.535628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.536057 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.536060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.536185 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.536734 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:26.536749 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:26.538656 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:26.540706 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.540716 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.540724 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.541028 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.541383 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.541387 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.541389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.541483 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.541891 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:26.541900 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:26.542025 (+ 125us) client_negotiation.cc:769] Sending connection context
0929 16:26:26.542173 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 16:26:26.542375 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":125,"threads_started":1}
I20260929 16:26:26.544175 6516 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:36225: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7bc65de05f0c4d39a89442538492b338"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 36225
}
attrs {
promote: true
}
}
dest_uuid: "6d732cddd2a8490ea5a12bdcc1798761"
cas_config_opid_index: -1
I20260929 16:26:26.544842 6516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260929 16:26:26.547112 7077 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7bc65de05f0c4d39a89442538492b338 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: true } } }
I20260929 16:26:26.547811 7022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:26.549932 7077 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:36225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260929 16:26:26.550068 7022 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 1 FOLLOWER]: Refusing update from remote peer 6d732cddd2a8490ea5a12bdcc1798761: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 16:26:26.551892 7076 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:26.555274 7078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6d732cddd2a8490ea5a12bdcc1798761" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: true } } }
I20260929 16:26:26.555441 7078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:26.557763 7022 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7bc65de05f0c4d39a89442538492b338 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: true } } }
I20260929 16:26:26.559092 7081 mvcc.cc:205] Tried to move back new op lower bound from 7334703863738716160 to 7334703863621099520. Current Snapshot: MvccSnapshot[applied={T|T < 7334703863662931968}]
I20260929 16:26:26.560086 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d732cddd2a8490ea5a12bdcc1798761. Latest consensus state: current_term: 1 leader_uuid: "6d732cddd2a8490ea5a12bdcc1798761" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: true } } }
I20260929 16:26:26.560202 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: This master's current role is: LEARNER
I20260929 16:26:26.560304 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d732cddd2a8490ea5a12bdcc1798761" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: true } } }
I20260929 16:26:26.560379 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: This master's current role is: LEARNER
I20260929 16:26:26.560453 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d732cddd2a8490ea5a12bdcc1798761" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: true } } }
I20260929 16:26:26.560518 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: This master's current role is: LEARNER
I20260929 16:26:26.560618 6980 master.cc:537] Master@127.0.98.61:36225 shutting down...
I20260929 16:26:26.565090 6980 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 1 LEARNER]: Raft consensus shutting down.
I20260929 16:26:26.565593 6980 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 1 LEARNER]: Raft consensus is shut down!
I20260929 16:26:26.565753 6980 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338: stopping tablet replica
I20260929 16:26:26.572388 6980 master.cc:552] Master@127.0.98.61:36225 shutdown complete.
I20260929 16:26:26.572466 6980 master_runner.cc:305] Clearing existing system tablet
I20260929 16:26:26.572714 6980 fs_manager.cc:359] Metadata directory not provided
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260929 16:26:26.572800 6980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:26.573777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33633
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44555,127.0.98.61:36225,127.0.98.60:33633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:26:26.574451 6980 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 16:26:26.575469 7090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:26.575676 6980 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:26.575830 6980 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 16:26:26.575866 6980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/wal
uuid: "7bc65de05f0c4d39a89442538492b338"
format_stamp: "Formatted at 2026-09-29 16:26:26 on dist-test-slave-vq9v"
server_key: "b0e6f620ac76759735ca29eb535e4eca"
server_key_iv: "9c26e34e54490442f58c9b2d3ecadcbc"
server_key_version: "encryptionkey@0"
I20260929 16:26:26.577488 6980 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:26.579430 6980 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 16:26:26.579545 6980 log.cc:1196] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 16:26:26.579820 6980 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338: Deleting consensus metadata
I20260929 16:26:26.580015 6980 master_runner.cc:315] Copying system tablet from 127.0.98.62:44555
I20260929 16:26:26.581187 6980 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:44555
I20260929 16:26:26.593391 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.585675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.0.98.62:44555)
0929 16:26:26.585818 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:26.585823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:26.586057 (+ 234us) server_negotiation.cc:407] Connection header received
0929 16:26:26.586210 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:26.586213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:26.586266 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:26.586357 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:26.586823 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.587346 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.588294 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.588522 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.588633 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.590025 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:26.590031 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:26.590035 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:26.590062 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:26.591668 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.592136 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.592139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.592141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.592230 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.592513 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.592516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.592518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.592710 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:26.592818 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:26.593083 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 16:26:26.593192 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:26:26.593425 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.585599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.0.1:43216)
0929 16:26:26.585950 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:26.585962 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:26.586107 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:26.586373 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:26.586376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:26.586591 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:26.586691 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.586722 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.587462 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.587467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.588153 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.588161 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.588641 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.588644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.588738 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.589428 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:26.589448 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:26.589867 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:26.591809 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.591812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.591814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.592029 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.592322 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.592327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.592329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.592416 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.592805 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:26.592808 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:26.592913 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 16:26:26.593019 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 16:26:26.593172 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":135,"threads_started":1}
I20260929 16:26:26.594439 6536 tablet_copy_service.cc:140] P 6d732cddd2a8490ea5a12bdcc1798761: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7bc65de05f0c4d39a89442538492b338 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43216)
I20260929 16:26:26.594619 6536 tablet_copy_service.cc:161] P 6d732cddd2a8490ea5a12bdcc1798761: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7bc65de05f0c4d39a89442538492b338 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43216: session id = 7bc65de05f0c4d39a89442538492b338-00000000000000000000000000000000
I20260929 16:26:26.596089 6536 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 16:26:26.597673 6980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 16:26:26.599855 6980 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338: tablet copy: Starting download of 0 data blocks...
I20260929 16:26:26.600068 6980 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338: tablet copy: Starting download of 1 WAL segments...
I20260929 16:26:26.602402 6980 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 16:26:26.605134 6536 tablet_copy_service.cc:342] P 6d732cddd2a8490ea5a12bdcc1798761: Request end of tablet copy session 7bc65de05f0c4d39a89442538492b338-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43216
I20260929 16:26:26.605369 6536 tablet_copy_service.cc:433] P 6d732cddd2a8490ea5a12bdcc1798761: ending tablet copy session 7bc65de05f0c4d39a89442538492b338-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7bc65de05f0c4d39a89442538492b338
I20260929 16:26:26.612586 6980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:26.614418 7097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:26.622664 6980 server_base.cc:1034] running on GCE node
W20260929 16:26:26.622779 7098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:26.623051 7100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:26.623364 6980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:26.623742 6980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:26.624907 6980 hybrid_clock.cc:648] HybridClock initialized: now 1790699186624914 us; error 59 us; skew 500 ppm
Sep 29 16:26:26 dist-test-slave-vq9v krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699186, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:26.626550 6980 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260929 16:26:26.627089 6980 webserver.cc:460] Webserver started at http://127.0.98.61:32887/ using document root <none> and password file <none>
I20260929 16:26:26.627306 6980 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:26.627362 6980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:26.628474 6980 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 16:26:26.629613 7107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:26.629856 6980 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 16:26:26.629954 6980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/wal
uuid: "7bc65de05f0c4d39a89442538492b338"
format_stamp: "Formatted at 2026-09-29 16:26:26 on dist-test-slave-vq9v"
server_key: "b0e6f620ac76759735ca29eb535e4eca"
server_key_iv: "9c26e34e54490442f58c9b2d3ecadcbc"
server_key_version: "encryptionkey@0"
I20260929 16:26:26.630064 6980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:26.638840 6980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:26.640961 6980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:26.646381 6980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36225
I20260929 16:26:26.647050 6980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-1/data/info.pb
I20260929 16:26:26.647454 7159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36225 every 8 connection(s)
I20260929 16:26:26.648650 7160 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:26:26.649698 7160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338: Bootstrap starting.
I20260929 16:26:26.656179 7160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:26:26.656522 7160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338: Bootstrap complete.
I20260929 16:26:26.657122 7160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: true } }
I20260929 16:26:26.657305 7160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7bc65de05f0c4d39a89442538492b338, State: Initialized, Role: LEARNER
I20260929 16:26:26.657430 7160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: true } }
I20260929 16:26:26.657766 7163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: true } } }
I20260929 16:26:26.657929 7163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: This master's current role is: LEARNER
I20260929 16:26:26.657939 7160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:26:26.661662 7174 catalog_manager.cc:1259] Loaded cluster ID: d3fbbee867e94a12b43864bbe215c0dc
I20260929 16:26:26.661748 7174 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338: loading cluster ID for follower catalog manager: success
I20260929 16:26:26.663280 7174 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338: acquiring CA information for follower catalog manager: success
I20260929 16:26:26.664495 7174 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 16:26:26.732623 7089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:26.732928 7089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:26.732971 7089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:26.737423 7089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:26:26.737502 7089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:26.737524 7089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:26.737541 7089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:26:26.737557 7089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:26:26.743139 7089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44555,127.0.98.61:36225,127.0.98.60:33633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:26.744361 7089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:26.745535 7089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:26.754006 7182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:26.754001 7179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:26.753998 7180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:26.754263 7089 server_base.cc:1034] running on GCE node
I20260929 16:26:26.755143 7089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:26.756621 7089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:26.757805 7089 hybrid_clock.cc:648] HybridClock initialized: now 1790699186757782 us; error 56 us; skew 500 ppm
Sep 29 16:26:26 dist-test-slave-vq9v krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699186, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:26.760887 7089 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260929 16:26:26.761986 7089 webserver.cc:460] Webserver started at http://127.0.98.60:39755/ using document root <none> and password file <none>
I20260929 16:26:26.762583 7089 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:26.762677 7089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:26.762913 7089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:26.765075 7089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/data/instance:
uuid: "fea1cb04b3954e36952f2144017d4d8c"
format_stamp: "Formatted at 2026-09-29 16:26:26 on dist-test-slave-vq9v"
server_key: "5a40033baf8fc8a0f474d0df2bbed6ca"
server_key_iv: "c24522c952746b4635641b5725ea76b3"
server_key_version: "encryptionkey@0"
I20260929 16:26:26.765756 7089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/wal/instance:
uuid: "fea1cb04b3954e36952f2144017d4d8c"
format_stamp: "Formatted at 2026-09-29 16:26:26 on dist-test-slave-vq9v"
server_key: "5a40033baf8fc8a0f474d0df2bbed6ca"
server_key_iv: "c24522c952746b4635641b5725ea76b3"
server_key_version: "encryptionkey@0"
I20260929 16:26:26.769863 7089 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:26:26.769997 6707 heartbeater.cc:503] Master 127.0.98.62:44555 was elected leader, sending a full tablet report...
I20260929 16:26:26.772827 7189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:26.774375 7089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:26.774524 7089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/wal
uuid: "fea1cb04b3954e36952f2144017d4d8c"
format_stamp: "Formatted at 2026-09-29 16:26:26 on dist-test-slave-vq9v"
server_key: "5a40033baf8fc8a0f474d0df2bbed6ca"
server_key_iv: "c24522c952746b4635641b5725ea76b3"
server_key_version: "encryptionkey@0"
I20260929 16:26:26.774652 7089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:26.789693 7089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:26.793069 7089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:26.793336 7089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:26.800393 7089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33633
I20260929 16:26:26.800869 7241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33633 every 8 connection(s)
I20260929 16:26:26.801952 7089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/data/info.pb
I20260929 16:26:26.804790 7242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:26.808393 7242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:26:26.812153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 7089
Sep 29 16:26:26 dist-test-slave-vq9v krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699186, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 16:26:26 dist-test-slave-vq9v krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 16:26:26.824087 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.810404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43224 (local address 127.0.98.62:44555)
0929 16:26:26.810557 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:26.810561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:26.811280 (+ 719us) server_negotiation.cc:407] Connection header received
0929 16:26:26.812544 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:26.812546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:26.812592 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:26.812668 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:26.814064 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.814684 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.816370 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.816709 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.816824 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.819734 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:26.819738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:26.819740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:26.819763 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:26.821511 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.822096 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.822102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.822105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.822209 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.822623 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.822627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.822630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.822872 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:26.822954 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:26.823754 (+ 800us) server_negotiation.cc:299] Negotiation successful
0929 16:26:26.823896 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:26:26.824940 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.810656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.0.1:43224)
0929 16:26:26.811105 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:26.811166 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:26.812303 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:26.812853 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:26.812861 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:26.813394 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:26.813907 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.813925 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.814845 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.814848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.816216 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.816227 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.816871 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.816875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.817029 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.817720 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:26.817749 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:26.819564 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:26.821648 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.821656 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.821664 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.821991 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.822307 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.822310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.822312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.822495 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.823002 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:26.823009 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:26.823399 (+ 390us) client_negotiation.cc:769] Sending connection context
0929 16:26:26.823688 (+ 289us) client_negotiation.cc:240] Negotiation successful
0929 16:26:26.824074 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":183,"thread_start_us":102,"threads_started":1}
I20260929 16:26:26.824939 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.813067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.0.98.60:33633)
0929 16:26:26.813403 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:26.813416 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:26.813437 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:26:26.813478 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:26.813482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:26.813629 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:26.813795 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:26.814562 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.815480 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.816841 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.817303 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.817429 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.820178 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:26.820185 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:26.820200 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:26.820226 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:26.822152 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.822657 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.822665 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.822678 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.822753 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.823137 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.823139 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.823140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.823429 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:26.823697 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:26.823896 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 16:26:26.824498 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":453,"server-negotiator.queue_time_us":230,"spinlock_wait_cycles":18816,"thread_start_us":153,"threads_started":1}
I20260929 16:26:26.826939 7242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:26:26 dist-test-slave-vq9v krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699186, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 16:26:26.835899 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.827479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36225 (local address 127.0.0.1:59460)
0929 16:26:26.827633 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:26.827646 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:26.827711 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:26.828209 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:26.828211 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:26.828371 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:26.828476 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.828481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.829748 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.829751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.830311 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.830319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.830921 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.830924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.831005 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.831519 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:26.831531 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:26.832925 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:26.834668 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.834671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.834673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.834860 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.835110 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.835113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.835114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.835191 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.835515 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:26.835516 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:26.835591 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 16:26:26.835667 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 16:26:26.835773 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 16:26:26.835945 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.827671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59460 (local address 127.0.98.61:36225)
0929 16:26:26.827921 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:26.827933 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:26.827945 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:26:26.827982 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:26.827984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:26.828026 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:26.828105 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:26.828606 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.829639 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.830479 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.830780 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.830887 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.833086 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:26.833090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:26.833092 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:26.833168 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:26.834566 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.834956 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.834959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.834960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.835033 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.835266 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.835268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.835269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.835439 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:26.835512 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:26.835726 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 16:26:26.835826 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":97,"threads_started":1}
I20260929 16:26:26.836824 7242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:26:26 dist-test-slave-vq9v krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699186, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 16:26:26.844507 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.837353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33633 (local address 127.0.0.1:43898)
0929 16:26:26.837436 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:26.837446 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:26.837532 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:26.837750 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:26.837752 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:26.837902 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:26.837961 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.837965 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.838668 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.838672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.839161 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.839168 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.839655 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.839658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.839725 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.840132 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:26.840142 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:26.841470 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:26.843056 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.843062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.843066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.843251 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.843508 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.843513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.843515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.843591 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.843936 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:26.843939 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:26.844015 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 16:26:26.844124 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 16:26:26.844292 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 16:26:26.844507 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.837450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43898 (local address 127.0.98.60:33633)
0929 16:26:26.837542 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:26.837545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:26.837559 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:26:26.837598 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:26.837604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:26.837638 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:26.837729 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:26.838061 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.838563 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.839309 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.839554 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.839659 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.841594 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:26.841597 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:26.841599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:26.841619 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:26.842951 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.843343 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.843348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.843350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.843422 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.843681 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.843684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.843686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.843844 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:26.843951 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:26.844182 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 16:26:26.844339 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":37}
I20260929 16:26:26.848069 7242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c: Bootstrap starting.
I20260929 16:26:26.850383 7242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:26.851038 7242 log.cc:824] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:26.852957 7242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c: No bootstrap required, opened a new log
I20260929 16:26:26.856235 7242 raft_consensus.cc:348] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } }
I20260929 16:26:26.856480 7242 raft_consensus.cc:374] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:26.856516 7242 raft_consensus.cc:729] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fea1cb04b3954e36952f2144017d4d8c, State: Initialized, Role: FOLLOWER
I20260929 16:26:26.856910 7242 consensus_queue.cc:260] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } }
I20260929 16:26:26.857404 7248 sys_catalog.cc:455] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } } }
I20260929 16:26:26.857535 7248 sys_catalog.cc:458] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:26:26.858047 7248 raft_consensus.cc:482] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:26:26.858176 7248 raft_consensus.cc:504] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } }
I20260929 16:26:26.858319 7242 sys_catalog.cc:564] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:26:26.859218 7248 leader_election.cc:288] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d732cddd2a8490ea5a12bdcc1798761 (127.0.98.62:44555), 7bc65de05f0c4d39a89442538492b338 (127.0.98.61:36225)
I20260929 16:26:26.859862 7134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fea1cb04b3954e36952f2144017d4d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bc65de05f0c4d39a89442538492b338" is_pre_election: true
I20260929 16:26:26.859862 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fea1cb04b3954e36952f2144017d4d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d732cddd2a8490ea5a12bdcc1798761" is_pre_election: true
I20260929 16:26:26.860095 7134 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 1 LEARNER]: Handling vote request from an unknown peer fea1cb04b3954e36952f2144017d4d8c
I20260929 16:26:26.860244 7134 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate fea1cb04b3954e36952f2144017d4d8c for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260929 16:26:26.860692 7191 leader_election.cc:302] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fea1cb04b3954e36952f2144017d4d8c; no voters: 6d732cddd2a8490ea5a12bdcc1798761, 7bc65de05f0c4d39a89442538492b338
I20260929 16:26:26.861310 7248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:26.862533 7248 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260929 16:26:26.863955 7259 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:26:26.864051 7259 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:26:26.867394 7089 master_runner.cc:428] Detected that this master fea1cb04b3954e36952f2144017d4d8c is joining an existing cluster
I20260929 16:26:26.867462 7089 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:33633
I20260929 16:26:26.878445 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 6578
I20260929 16:26:26.883996 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.870083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43908 (local address 127.0.98.60:33633)
0929 16:26:26.870218 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:26.870223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:26.871263 (+ 1040us) server_negotiation.cc:407] Connection header received
0929 16:26:26.871395 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:26.871400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:26.871436 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:26.871482 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:26.872031 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.872772 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.874045 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.874363 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.874436 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.880074 (+ 5638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:26.880078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:26.880081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:26.880118 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:26.881988 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.882506 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.882510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.882512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.882607 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.882931 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.882935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.882937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.883193 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:26.883297 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:26.883619 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 16:26:26.883802 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":56}
I20260929 16:26:26.884038 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.870571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33633 (local address 127.0.0.1:43908)
0929 16:26:26.871198 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:26.871211 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:26.871312 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:26.871610 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:26.871613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:26.871803 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:26.871935 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.871943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.872892 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.872895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.873924 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.873935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.874576 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.874579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.874825 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.879358 (+ 4533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:26.879379 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:26.879927 (+ 548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:26.882107 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.882114 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.882119 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.882400 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.882713 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.882718 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.882723 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.882827 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.883305 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:26.883311 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:26.883429 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 16:26:26.883547 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 16:26:26.883723 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":173,"spinlock_wait_cycles":2432,"thread_start_us":114,"threads_started":1}
I20260929 16:26:26.886475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41403
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44555
--webserver_port=43521
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:44555,127.0.98.61:36225
--tserver_master_addrs=127.0.98.62:44555,127.0.98.61:36225,127.0.98.60:33633 with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:26:26.899124 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.870091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59468 (local address 127.0.98.61:36225)
0929 16:26:26.870269 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:26.870273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:26.870944 (+ 671us) server_negotiation.cc:407] Connection header received
0929 16:26:26.871070 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:26.871073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:26.871126 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:26.871204 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:26.871940 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.872835 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.874229 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.874637 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.874719 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.894881 (+ 20162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:26.894886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:26.894889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:26.894922 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:26.897044 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.897780 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.897784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.897786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.897871 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.898165 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.898169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.898181 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.898408 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:26.898486 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:26.898809 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 16:26:26.898950 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:26:26.899541 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.869944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.0.1:43238)
0929 16:26:26.870584 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:26.870597 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:26.870736 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:26.871329 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:26.871335 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:26.871616 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:26.871806 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.871812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.872952 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.872954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.873936 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.873942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.874576 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.874579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.874822 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.893938 (+ 19116us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:26.893967 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:26.894718 (+ 751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:26.897240 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.897247 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.897253 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.897720 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.898141 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.898147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.898153 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.898258 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.898869 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:26.898876 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:26.899000 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 16:26:26.899134 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 16:26:26.899330 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":120,"spinlock_wait_cycles":1408,"thread_start_us":125,"threads_started":1}
I20260929 16:26:26.900219 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.870385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43238 (local address 127.0.98.62:44555)
0929 16:26:26.870513 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:26.870518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:26.870669 (+ 151us) server_negotiation.cc:407] Connection header received
0929 16:26:26.870882 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:26.870886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:26.870943 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:26.871040 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:26.871955 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.872797 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.874055 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.874383 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.874449 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.894966 (+ 20517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:26.894972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:26.894977 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:26.895022 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:26.896956 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.897872 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.897879 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.897881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.897993 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.898391 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.898398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.898403 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.898683 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:26.898812 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:26.899902 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0929 16:26:26.900038 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:26:26.900476 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.869944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36225 (local address 127.0.0.1:59468)
0929 16:26:26.870855 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:26.870868 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:26.870973 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:26.871225 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:26.871229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:26.871550 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:26.871763 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.871772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.873011 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.873015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.874067 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.874077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.875535 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.875543 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.875704 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.894031 (+ 18327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:26.894049 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:26.894653 (+ 604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:26.897235 (+ 2582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.897239 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.897241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.897654 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.897982 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.897985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.897987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.898069 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.898540 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:26.898543 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:26.898637 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 16:26:26.899920 (+ 1283us) client_negotiation.cc:240] Negotiation successful
0929 16:26:26.900073 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":384,"thread_start_us":123,"threads_started":1}
Sep 29 16:26:26 dist-test-slave-vq9v krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699185, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 16:26:26.914850 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.904511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33633 (local address 127.0.0.1:43912)
0929 16:26:26.904739 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:26.904755 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:26.904849 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:26.905159 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:26.905162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:26.905434 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:26.905552 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.905560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.906579 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.906587 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.907626 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.907638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.908212 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.908216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.908319 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.908852 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:26.908867 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:26.910882 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:26.913014 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.913018 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.913020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.913437 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.913725 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.913728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.913731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.913829 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.914290 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:26.914295 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:26.914411 (+ 116us) client_negotiation.cc:769] Sending connection context
0929 16:26:26.914517 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 16:26:26.914675 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260929 16:26:26.914867 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.904606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43912 (local address 127.0.98.60:33633)
0929 16:26:26.904720 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:26.904724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:26.904803 (+ 79us) server_negotiation.cc:407] Connection header received
0929 16:26:26.904978 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:26.904981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:26.905035 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:26.905116 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:26.905678 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.906451 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.907753 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.908094 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.908200 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.911042 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:26.911046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:26.911049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:26.911076 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:26.912876 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.913531 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.913535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.913537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.913622 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.913934 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.913937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.913939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.914149 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:26.914275 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:26.914599 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 16:26:26.914714 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 16:26:26.916394 6516 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:33633: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 33633
}
attrs {
promote: true
}
}
dest_uuid: "6d732cddd2a8490ea5a12bdcc1798761"
cas_config_opid_index: 5
I20260929 16:26:26.917022 6516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: true } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: true } }
I20260929 16:26:26.918354 7076 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fea1cb04b3954e36952f2144017d4d8c (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: true } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: true } } }
I20260929 16:26:26.919792 7076 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:33633
I20260929 16:26:26.919965 7078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6d732cddd2a8490ea5a12bdcc1798761" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: true } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: true } } }
I20260929 16:26:26.920068 7078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:26.920497 7089 master.cc:537] Master@127.0.98.60:33633 shutting down...
I20260929 16:26:26.920832 7216 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 1 FOLLOWER]: Refusing update from remote peer 6d732cddd2a8490ea5a12bdcc1798761: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 16:26:26.921451 7078 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 16:26:26.922904 6502 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260929 16:26:26.927397 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.918755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59480 (local address 127.0.98.61:36225)
0929 16:26:26.918903 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:26.918908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:26.919183 (+ 275us) server_negotiation.cc:407] Connection header received
0929 16:26:26.919237 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:26.919240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:26.919299 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:26.919385 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:26.921238 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.924036 (+ 2798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.925749 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.926436 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.926549 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.926711 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:26.927171 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 16:26:26.927256 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:26:26.927824 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.918516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36225 (local address 127.0.0.1:59480)
0929 16:26:26.918986 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:26.919000 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:26.919084 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:26.919506 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:26.919509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:26.919526 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:26.919615 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.919621 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.924200 (+ 4579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.924204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.925589 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.925601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.926778 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.926783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.926880 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.926926 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 16:26:26.927612 (+ 686us) client_negotiation.cc:240] Negotiation successful
0929 16:26:26.927697 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406}
I20260929 16:26:26.928354 7134 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 1 LEARNER]: Refusing update from remote peer 6d732cddd2a8490ea5a12bdcc1798761: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 16:26:26.928800 7078 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 16:26:26.929965 7134 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fea1cb04b3954e36952f2144017d4d8c (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: true } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: true } } }
I20260929 16:26:26.932020 7163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d732cddd2a8490ea5a12bdcc1798761. Latest consensus state: current_term: 1 leader_uuid: "6d732cddd2a8490ea5a12bdcc1798761" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: true } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: true } } }
I20260929 16:26:26.932181 7163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: This master's current role is: LEARNER
I20260929 16:26:26.932292 7163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d732cddd2a8490ea5a12bdcc1798761" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: true } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: true } } }
I20260929 16:26:26.932361 7163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: This master's current role is: LEARNER
I20260929 16:26:26.933440 7089 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:26.933705 7078 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761: attempting to promote NON_VOTER 7bc65de05f0c4d39a89442538492b338 to VOTER
I20260929 16:26:26.933857 7089 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:26.934120 7089 tablet_replica.cc:330] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c: stopping tablet replica
I20260929 16:26:26.934185 7078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: true } }
I20260929 16:26:26.936183 7134 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 1 LEARNER]: Refusing update from remote peer 6d732cddd2a8490ea5a12bdcc1798761: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 16:26:26.936650 7077 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260929 16:26:26.936868 6502 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 -> Peer fea1cb04b3954e36952f2144017d4d8c (127.0.98.60:33633): Couldn't send request to peer fea1cb04b3954e36952f2144017d4d8c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 16:26:26.939090 7077 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7bc65de05f0c4d39a89442538492b338 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: true } } }
I20260929 16:26:26.940629 7077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6d732cddd2a8490ea5a12bdcc1798761" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: true } } }
I20260929 16:26:26.940749 7077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:26.940872 7077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d732cddd2a8490ea5a12bdcc1798761" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: true } } }
I20260929 16:26:26.940941 7077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:26.941566 7089 master.cc:552] Master@127.0.98.60:33633 shutdown complete.
I20260929 16:26:26.941620 7089 master_runner.cc:305] Clearing existing system tablet
I20260929 16:26:26.941867 7089 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:26.941915 7089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:26.941784 7134 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7bc65de05f0c4d39a89442538492b338 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: true } } }
I20260929 16:26:26.943286 7271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d732cddd2a8490ea5a12bdcc1798761" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: true } } }
I20260929 16:26:26.943418 7271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:26:26.944432 7089 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:26:26.945636 7273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:26.945921 7089 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:26.946056 7089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 16:26:26.946092 7089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/wal
uuid: "fea1cb04b3954e36952f2144017d4d8c"
format_stamp: "Formatted at 2026-09-29 16:26:26 on dist-test-slave-vq9v"
server_key: "5a40033baf8fc8a0f474d0df2bbed6ca"
server_key_iv: "c24522c952746b4635641b5725ea76b3"
server_key_version: "encryptionkey@0"
I20260929 16:26:26.948276 7089 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:26.951093 7089 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 16:26:26.951216 7089 log.cc:1196] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 16:26:26.951524 7089 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c: Deleting consensus metadata
I20260929 16:26:26.951742 7089 master_runner.cc:315] Copying system tablet from 127.0.98.62:44555
I20260929 16:26:26.953365 7089 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:44555
I20260929 16:26:26.964473 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.954108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43248 (local address 127.0.98.62:44555)
0929 16:26:26.954250 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:26.954255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:26.954454 (+ 199us) server_negotiation.cc:407] Connection header received
0929 16:26:26.954581 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:26.954585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:26.954644 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:26.954762 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:26.955440 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.956379 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.957848 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.958243 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.958334 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.960098 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:26.960103 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:26.960106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:26.960137 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:26.962239 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.962809 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.962813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.962816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.962966 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:26.963305 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:26.963311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:26.963315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:26.963594 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:26.963706 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:26.964151 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 16:26:26.964307 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:26:26.964473 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:26.953999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.0.1:43248)
0929 16:26:26.954340 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:26.954357 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:26.954484 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:26.954904 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:26.954908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:26.955146 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:26.955276 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.955285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.956554 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.956558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.957664 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:26.957677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:26.958533 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:26.958537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:26.958649 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:26.959367 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:26.959395 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:26.959913 (+ 518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:26.962377 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.962381 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.962384 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.962679 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.963079 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:26.963082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:26.963085 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:26.963190 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:26.963717 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:26.963720 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:26.963931 (+ 211us) client_negotiation.cc:769] Sending connection context
0929 16:26:26.964065 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 16:26:26.964238 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":105,"threads_started":1}
I20260929 16:26:26.965530 6536 tablet_copy_service.cc:140] P 6d732cddd2a8490ea5a12bdcc1798761: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fea1cb04b3954e36952f2144017d4d8c ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43248)
I20260929 16:26:26.965659 6536 tablet_copy_service.cc:161] P 6d732cddd2a8490ea5a12bdcc1798761: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fea1cb04b3954e36952f2144017d4d8c at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43248: session id = fea1cb04b3954e36952f2144017d4d8c-00000000000000000000000000000000
I20260929 16:26:26.966881 6536 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 16:26:26.968328 7089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 16:26:26.970916 7089 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c: tablet copy: Starting download of 0 data blocks...
I20260929 16:26:26.971122 7089 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c: tablet copy: Starting download of 1 WAL segments...
I20260929 16:26:26.972839 7089 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 16:26:26.975054 6536 tablet_copy_service.cc:342] P 6d732cddd2a8490ea5a12bdcc1798761: Request end of tablet copy session fea1cb04b3954e36952f2144017d4d8c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43248
I20260929 16:26:26.975215 6536 tablet_copy_service.cc:433] P 6d732cddd2a8490ea5a12bdcc1798761: ending tablet copy session fea1cb04b3954e36952f2144017d4d8c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fea1cb04b3954e36952f2144017d4d8c
I20260929 16:26:26.983824 7089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:26.985773 7280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:26.985816 7281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:26.986045 7283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:26.986653 7089 server_base.cc:1034] running on GCE node
I20260929 16:26:26.986828 7089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:26.987138 7089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:26.988291 7089 hybrid_clock.cc:648] HybridClock initialized: now 1790699186988266 us; error 43 us; skew 500 ppm
Sep 29 16:26:26 dist-test-slave-vq9v krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699186, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:26.990103 7089 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260929 16:26:26.990607 7089 webserver.cc:460] Webserver started at http://127.0.98.60:41259/ using document root <none> and password file <none>
I20260929 16:26:26.990825 7089 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:26.990871 7089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:26.993580 7089 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260929 16:26:26.994663 7290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:26.994877 7089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 16:26:26.994926 7089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/wal
uuid: "fea1cb04b3954e36952f2144017d4d8c"
format_stamp: "Formatted at 2026-09-29 16:26:26 on dist-test-slave-vq9v"
server_key: "5a40033baf8fc8a0f474d0df2bbed6ca"
server_key_iv: "c24522c952746b4635641b5725ea76b3"
server_key_version: "encryptionkey@0"
I20260929 16:26:26.995007 7089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:27.018374 6841 heartbeater.cc:503] Master 127.0.98.62:44555 was elected leader, sending a full tablet report...
I20260929 16:26:27.028956 7089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:27.035117 7089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:27.044237 7089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33633
I20260929 16:26:27.044260 7345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33633 every 8 connection(s)
I20260929 16:26:27.044914 7089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/master-2/data/info.pb
I20260929 16:26:27.047582 7346 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:26:27.048914 7346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c: Bootstrap starting.
W20260929 16:26:27.051844 7268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:27.052237 7268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:27.052330 7268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:27.059042 7268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:27.059124 7268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:27.059219 7268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:26:27.065070 7346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:26:27.065471 7346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c: Bootstrap complete.
I20260929 16:26:27.066295 7346 raft_consensus.cc:348] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: true } }
I20260929 16:26:27.066469 7346 raft_consensus.cc:729] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fea1cb04b3954e36952f2144017d4d8c, State: Initialized, Role: LEARNER
I20260929 16:26:27.066622 7346 consensus_queue.cc:260] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: true } }
I20260929 16:26:27.067014 7349 sys_catalog.cc:455] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: true } } }
I20260929 16:26:27.067140 7349 sys_catalog.cc:458] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [sys.catalog]: This master's current role is: LEARNER
I20260929 16:26:27.067183 7346 sys_catalog.cc:564] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:26:27.067353 7268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43521
--tserver_master_addrs=127.0.98.62:44555,127.0.98.61:36225,127.0.98.60:33633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:27.068704 7268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:27.070045 7268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:26:27.070482 7360 catalog_manager.cc:1259] Loaded cluster ID: d3fbbee867e94a12b43864bbe215c0dc
I20260929 16:26:27.070528 7360 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c: loading cluster ID for follower catalog manager: success
I20260929 16:26:27.072187 7360 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c: acquiring CA information for follower catalog manager: success
I20260929 16:26:27.073349 7360 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 16:26:27.083488 7365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:27.083488 7362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:27.083510 7363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:27.083999 7268 server_base.cc:1034] running on GCE node
I20260929 16:26:27.084494 7268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:27.085244 7268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:27.086462 7268 hybrid_clock.cc:648] HybridClock initialized: now 1790699187086440 us; error 39 us; skew 500 ppm
Sep 29 16:26:27 dist-test-slave-vq9v krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699187, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:27.089376 7268 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 16:26:27.090379 7268 webserver.cc:460] Webserver started at http://127.0.98.1:43521/ using document root <none> and password file <none>
I20260929 16:26:27.090982 7268 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:27.091077 7268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:27.095233 7268 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:26:27.097488 7372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:27.098853 7268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:26:27.099026 7268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "13adcbd1e74248ab871d13068c0f23c9"
format_stamp: "Formatted at 2026-09-29 16:26:25 on dist-test-slave-vq9v"
server_key: "c368fc09d1ba5da71283a3bb37d57b02"
server_key_iv: "2cd8580d4e56f07867c26b816955c94c"
server_key_version: "encryptionkey@0"
I20260929 16:26:27.099624 7268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:27.122383 7268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:27.125733 7268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:27.125937 7268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:27.126531 7268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:27.127398 7268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:27.127470 7268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:27.127540 7268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:27.127593 7268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:27.138111 7268 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41403
I20260929 16:26:27.138132 7485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41403 every 8 connection(s)
I20260929 16:26:27.139206 7268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:26:27.149585 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 7268
I20260929 16:26:27.149786 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 6712
Sep 29 16:26:27 dist-test-slave-vq9v krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699187, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 29 16:26:27 dist-test-slave-vq9v krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699187, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 16:26:27 dist-test-slave-vq9v krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699187, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 16:26:27.156759 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.142887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58379 (local address 127.0.98.60:33633)
0929 16:26:27.143123 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.143135 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.143150 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:26:27.143878 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.143882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.143994 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.144069 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:27.145562 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.146604 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.147678 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.147953 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.148074 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.151794 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:27.151798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:27.151801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:27.151848 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:27.153822 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.154505 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.154511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.154517 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.154614 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.155053 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.155056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.155061 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.155570 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:27.155672 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.156393 (+ 721us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.156533 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":86,"threads_started":1}
I20260929 16:26:27.157608 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.141852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59443 (local address 127.0.98.62:44555)
0929 16:26:27.142125 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.142130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.142899 (+ 769us) server_negotiation.cc:407] Connection header received
0929 16:26:27.143878 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.143882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.143935 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.144016 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:27.145562 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.146379 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.148042 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.148354 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.148456 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.152896 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:27.152900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:27.152902 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:27.152933 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:27.155273 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.155929 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.155932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.155934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.156017 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.156299 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.156304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.156308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.156542 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:27.156631 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.157364 (+ 733us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.157475 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260929 16:26:27.157624 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.141411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55421 (local address 127.0.98.61:36225)
0929 16:26:27.142026 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.142033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.143289 (+ 1256us) server_negotiation.cc:407] Connection header received
0929 16:26:27.143785 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.143789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.143855 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.143973 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:27.145562 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.146218 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.148568 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.148870 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.148992 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.153273 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:27.153281 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:27.153287 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:27.153325 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:27.155230 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.155962 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.155969 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.155975 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.156072 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.156422 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.156429 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.156434 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.156694 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:27.156786 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.157175 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.157359 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502}
I20260929 16:26:27.158506 7490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.141722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36225 (local address 127.0.98.1:55421)
0929 16:26:27.143189 (+ 1467us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:27.143201 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:27.143571 (+ 370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:27.144159 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:27.144166 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:27.144750 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:27.145399 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.145415 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.146362 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.146366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.147911 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.147920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.148994 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.148999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.149226 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.150763 (+ 1537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:27.150786 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:27.152764 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:27.155369 (+ 2605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:27.155376 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:27.155382 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:27.155835 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:27.156204 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:27.156210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:27.156216 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:27.156315 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:27.156849 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:27.156855 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:27.156958 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 16:26:27.157815 (+ 857us) client_negotiation.cc:240] Negotiation successful
0929 16:26:27.158005 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1385,"mutex_wait_us":352,"spinlock_wait_cycles":113280,"thread_start_us":155,"threads_started":1}
I20260929 16:26:27.158777 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.141722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.1:59443)
0929 16:26:27.142813 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:27.142826 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:27.143556 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:27.144161 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:27.144168 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:27.144802 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:27.145400 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.145417 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.146494 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.146497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.147909 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.147920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.148477 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.148480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.148551 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.149370 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:27.149403 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:27.152564 (+ 3161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:27.155362 (+ 2798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:27.155368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:27.155370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:27.155825 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:27.156114 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:27.156117 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:27.156118 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:27.156193 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:27.156675 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:27.156678 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:27.156773 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 16:26:27.157166 (+ 393us) client_negotiation.cc:240] Negotiation successful
0929 16:26:27.157416 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"mutex_wait_us":352,"spinlock_wait_cycles":93056,"thread_start_us":156,"threads_started":1}
I20260929 16:26:27.159525 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.141722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33633 (local address 127.0.98.1:58379)
0929 16:26:27.142412 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:27.142489 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:27.143556 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:27.144175 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:27.144199 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:27.144802 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:27.145400 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.145417 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.146708 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.146712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.147547 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.147556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.148239 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.148242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.148446 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.149370 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:27.149398 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:27.151483 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:27.153974 (+ 2491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:27.153983 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:27.154002 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:27.154380 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:27.154733 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:27.154739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:27.154744 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:27.154927 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:27.155729 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:27.155736 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:27.156080 (+ 344us) client_negotiation.cc:769] Sending connection context
0929 16:26:27.156954 (+ 874us) client_negotiation.cc:240] Negotiation successful
0929 16:26:27.159343 (+ 2389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":53,"spinlock_wait_cycles":11264,"thread_start_us":760,"threads_started":1}
I20260929 16:26:27.159879 7488 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36225
I20260929 16:26:27.160108 7488 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:27.159894 7487 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44555
I20260929 16:26:27.160187 7487 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:27.160315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43549
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44555
--webserver_port=45099
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:44555,127.0.98.61:36225
--tserver_master_addrs=127.0.98.62:44555,127.0.98.61:36225,127.0.98.60:33633 with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:26:27.160703 7487 heartbeater.cc:511] Master 127.0.98.62:44555 requested a full tablet report, sending...
I20260929 16:26:27.160976 7486 heartbeater.cc:347] Connected to a master server at 127.0.98.60:33633
I20260929 16:26:27.161043 7486 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:27.161185 7486 heartbeater.cc:511] Master 127.0.98.60:33633 requested a full tablet report, sending...
I20260929 16:26:27.161862 7488 heartbeater.cc:511] Master 127.0.98.61:36225 requested a full tablet report, sending...
I20260929 16:26:27.162446 7306 ts_manager.cc:194] Registered new tserver with Master: 13adcbd1e74248ab871d13068c0f23c9 (127.0.98.1:41403)
I20260929 16:26:27.163069 7124 ts_manager.cc:194] Registered new tserver with Master: 13adcbd1e74248ab871d13068c0f23c9 (127.0.98.1:41403)
I20260929 16:26:27.163368 6516 ts_manager.cc:194] Re-registered known tserver with Master: 13adcbd1e74248ab871d13068c0f23c9 (127.0.98.1:41403)
I20260929 16:26:27.164009 6516 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59443
I20260929 16:26:27.241914 6975 heartbeater.cc:503] Master 127.0.98.62:44555 was elected leader, sending a full tablet report...
W20260929 16:26:27.278429 7494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:27.278702 7494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:27.278755 7494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:27.282872 7494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:27.282945 7494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:27.283005 7494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:26:27.288511 7494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45099
--tserver_master_addrs=127.0.98.62:44555,127.0.98.61:36225,127.0.98.60:33633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:27.289738 7494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:27.290879 7494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:26:27.301560 7494 server_base.cc:1034] running on GCE node
W20260929 16:26:27.301505 7499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:27.301505 7500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:27.301505 7502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:27.302342 7494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:27.303139 7494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:27.304346 7494 hybrid_clock.cc:648] HybridClock initialized: now 1790699187304311 us; error 55 us; skew 500 ppm
Sep 29 16:26:27 dist-test-slave-vq9v krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699187, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:27.307605 7494 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 16:26:27.308902 7494 webserver.cc:460] Webserver started at http://127.0.98.2:45099/ using document root <none> and password file <none>
I20260929 16:26:27.309808 7494 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:27.309900 7494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:27.314433 7494 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:26:27.316807 7509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:27.318336 7494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:26:27.318455 7494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "bcbdbf5a43df466cb16a8911c91e24a6"
format_stamp: "Formatted at 2026-09-29 16:26:25 on dist-test-slave-vq9v"
server_key: "58297428cc5bad00e91a09c94c3d32d3"
server_key_iv: "11295fe37359087bf82523a53aba285d"
server_key_version: "encryptionkey@0"
I20260929 16:26:27.319092 7494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:27.343778 7494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:27.347457 7494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:27.347683 7494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:27.348371 7494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:27.349313 7494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:27.349362 7494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:27.349404 7494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:27.349457 7494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:27.359300 7494 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43549
I20260929 16:26:27.359325 7622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43549 every 8 connection(s)
I20260929 16:26:27.360433 7494 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:26:27.369771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 7494
I20260929 16:26:27.369969 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 6846
Sep 29 16:26:27 dist-test-slave-vq9v krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699187, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 29 16:26:27 dist-test-slave-vq9v krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699187, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 16:26:27 dist-test-slave-vq9v krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699187, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 16:26:27.377161 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46539
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44555
--webserver_port=44219
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:44555,127.0.98.61:36225
--tserver_master_addrs=127.0.98.62:44555,127.0.98.61:36225,127.0.98.60:33633 with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:26:27.378984 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.362474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50371 (local address 127.0.98.61:36225)
0929 16:26:27.362590 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.362594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.364131 (+ 1537us) server_negotiation.cc:407] Connection header received
0929 16:26:27.364907 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.364911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.364968 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.365070 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:27.366488 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.367328 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.369326 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.369664 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.369754 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.374124 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:27.374129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:27.374134 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:27.374164 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:27.375994 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.376904 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.376908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.376910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.376996 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.377466 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.377470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.377471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.377752 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:27.377850 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.378651 (+ 801us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.378797 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:26:27.380067 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.362328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43247 (local address 127.0.98.60:33633)
0929 16:26:27.362507 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.362513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.363943 (+ 1430us) server_negotiation.cc:407] Connection header received
0929 16:26:27.364915 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.364918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.364985 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.365078 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:27.367499 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.368135 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.369336 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.369650 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.369771 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.373827 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:27.373831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:27.373835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:27.373870 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:27.376102 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.376904 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.376908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.376910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.376996 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.379032 (+ 2036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.379038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.379039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.379264 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:27.379351 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.379781 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.379914 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:26:27.380697 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.363290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51709 (local address 127.0.98.62:44555)
0929 16:26:27.363436 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.363455 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.365246 (+ 1791us) server_negotiation.cc:407] Connection header received
0929 16:26:27.365412 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.365415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.365483 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.365553 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:27.366542 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.367364 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.369326 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.369680 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.369753 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.376804 (+ 7051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:27.376808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:27.376811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:27.376840 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:27.378631 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.379201 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.379205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.379206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.379309 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.379641 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.379645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.379647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.379878 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:27.379969 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.380341 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.380460 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:26:27.380712 7628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.362850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36225 (local address 127.0.98.2:50371)
0929 16:26:27.364025 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:27.364037 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:27.364691 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:27.365279 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:27.365286 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:27.365840 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:27.366347 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.366361 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.367539 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.367543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.369151 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.369162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.369977 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.369983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.371494 (+ 1511us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.372011 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:27.372031 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:27.373950 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:27.376143 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:27.376153 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:27.376241 (+ 88us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:27.376782 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:27.377112 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:27.377116 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:27.377119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:27.377351 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:27.377923 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:27.377933 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:27.378342 (+ 409us) client_negotiation.cc:769] Sending connection context
0929 16:26:27.379181 (+ 839us) client_negotiation.cc:240] Negotiation successful
0929 16:26:27.379571 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"mutex_wait_us":274,"spinlock_wait_cycles":21504,"thread_start_us":165,"threads_started":1}
I20260929 16:26:27.380710 7627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.362850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33633 (local address 127.0.98.2:43247)
0929 16:26:27.363781 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:27.363836 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:27.364719 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:27.365275 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:27.365285 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:27.365951 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:27.366375 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.366384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.368247 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.368252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.369124 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.369133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.369978 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.369985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.371193 (+ 1208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.371860 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:27.371887 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:27.373580 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:27.376392 (+ 2812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:27.376395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:27.376397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:27.376781 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:27.378788 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:27.378791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:27.378793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:27.378918 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:27.379481 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:27.379483 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:27.379578 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 16:26:27.380262 (+ 684us) client_negotiation.cc:240] Negotiation successful
0929 16:26:27.380403 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":115,"spinlock_wait_cycles":2816,"thread_start_us":708,"threads_started":1}
I20260929 16:26:27.381521 7629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.363189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.2:51709)
0929 16:26:27.365116 (+ 1927us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:27.365130 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:27.365275 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:27.365720 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:27.365723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:27.365990 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:27.366377 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.366384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.367539 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.367543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.369151 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.369162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.369997 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.370000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.373920 (+ 3920us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.374879 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:27.374899 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:27.376614 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:27.378767 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:27.378773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:27.378776 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:27.379065 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:27.379428 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:27.379432 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:27.379434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:27.379525 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:27.380029 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:27.380035 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:27.380140 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 16:26:27.380859 (+ 719us) client_negotiation.cc:240] Negotiation successful
0929 16:26:27.381038 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1862,"mutex_wait_us":184,"spinlock_wait_cycles":2432,"thread_start_us":81,"threads_started":1}
I20260929 16:26:27.382046 7625 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36225
I20260929 16:26:27.382301 7625 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:27.382887 7625 heartbeater.cc:511] Master 127.0.98.61:36225 requested a full tablet report, sending...
I20260929 16:26:27.383503 7623 heartbeater.cc:347] Connected to a master server at 127.0.98.60:33633
I20260929 16:26:27.383599 7623 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:27.383805 7623 heartbeater.cc:511] Master 127.0.98.60:33633 requested a full tablet report, sending...
I20260929 16:26:27.384032 7624 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44555
I20260929 16:26:27.384104 7624 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:27.384284 7624 heartbeater.cc:511] Master 127.0.98.62:44555 requested a full tablet report, sending...
I20260929 16:26:27.384289 7124 ts_manager.cc:194] Registered new tserver with Master: bcbdbf5a43df466cb16a8911c91e24a6 (127.0.98.2:43549)
I20260929 16:26:27.384773 7306 ts_manager.cc:194] Registered new tserver with Master: bcbdbf5a43df466cb16a8911c91e24a6 (127.0.98.2:43549)
I20260929 16:26:27.384935 6516 ts_manager.cc:194] Re-registered known tserver with Master: bcbdbf5a43df466cb16a8911c91e24a6 (127.0.98.2:43549)
I20260929 16:26:27.385582 6516 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51709
I20260929 16:26:27.390357 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.386982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43924 (local address 127.0.98.60:33633)
0929 16:26:27.387098 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.387102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.387121 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:26:27.387159 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.387161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.387265 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.387374 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:27.387609 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.388161 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.389164 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.389814 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.389952 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.390122 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.390159 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.390239 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:26:27.390357 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.386833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33633 (local address 127.0.0.1:43924)
0929 16:26:27.386954 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:27.386970 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:27.387052 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:27.387383 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:27.387386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:27.387402 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:27.387496 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.387502 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.388277 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.388283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.389028 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.389037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.389953 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.389956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.390043 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.390064 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:26:27.390130 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 16:26:27.390215 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 16:26:27.391160 7311 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 16:26:27.391709 7349 sys_catalog.cc:455] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d732cddd2a8490ea5a12bdcc1798761. Latest consensus state: current_term: 1 leader_uuid: "6d732cddd2a8490ea5a12bdcc1798761" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: true } } }
I20260929 16:26:27.391871 7349 sys_catalog.cc:458] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [sys.catalog]: This master's current role is: LEARNER
W20260929 16:26:27.498291 7630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:27.498540 7630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:27.498584 7630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:27.502727 7630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:27.502789 7630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:27.502851 7630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:26:27.508332 7630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46539
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44219
--tserver_master_addrs=127.0.98.62:44555,127.0.98.61:36225,127.0.98.60:33633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:27.509443 7630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:27.510408 7630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:27.520282 7635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:27.520355 7630 server_base.cc:1034] running on GCE node
W20260929 16:26:27.520306 7636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:27.520282 7638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:27.520944 7630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:27.521652 7630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:27.522823 7630 hybrid_clock.cc:648] HybridClock initialized: now 1790699187522805 us; error 31 us; skew 500 ppm
Sep 29 16:26:27 dist-test-slave-vq9v krb5kdc[6471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699187, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:27.525920 7630 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 16:26:27.526901 7630 webserver.cc:460] Webserver started at http://127.0.98.3:44219/ using document root <none> and password file <none>
I20260929 16:26:27.527526 7630 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:27.527602 7630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:27.531736 7630 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:26:27.534153 7645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:27.535538 7630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:26:27.535681 7630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "4bcbbb5789854afbbea071a979eccfb7"
format_stamp: "Formatted at 2026-09-29 16:26:26 on dist-test-slave-vq9v"
server_key: "68f0b8011771afd7a384a7a6e0667374"
server_key_iv: "809f34e05188ab3f9d1369c70c07e04b"
server_key_version: "encryptionkey@0"
I20260929 16:26:27.536322 7630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:27.550488 7630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:27.553979 7630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:27.554205 7630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:27.554834 7630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:27.555843 7630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:27.555922 7630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:27.555989 7630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:27.556041 7630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:27.565675 7630 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46539
I20260929 16:26:27.565702 7758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46539 every 8 connection(s)
I20260929 16:26:27.566785 7630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:26:27.576905 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 7630
Sep 29 16:26:27 dist-test-slave-vq9v krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699187, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 16:26:27 dist-test-slave-vq9v krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699187, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 16:26:27 dist-test-slave-vq9v krb5kdc[6471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699187, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 16:26:27.584580 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.570336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51863 (local address 127.0.98.62:44555)
0929 16:26:27.570469 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.570473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.570489 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:26:27.571121 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.571125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.571178 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.571231 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:27.572650 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.573490 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.574509 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.574728 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.574848 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.580401 (+ 5553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:27.580408 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:27.580413 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:27.580444 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:27.582306 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.582954 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.582957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.582959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.583043 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.583361 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.583363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.583365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.583870 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:27.583963 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.584278 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.584409 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:26:27.584650 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.568940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43773 (local address 127.0.98.61:36225)
0929 16:26:27.569049 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.569053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.570599 (+ 1546us) server_negotiation.cc:407] Connection header received
0929 16:26:27.571106 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.571109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.571167 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.571270 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:27.572634 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.573166 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.574926 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.575275 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.575367 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.579518 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:27.579522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:27.579525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:27.579557 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:27.582272 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.582954 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.582957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.582959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.583043 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.583359 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.583362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.583364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.583563 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:27.583676 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.584402 (+ 726us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.584511 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 16:26:27.585691 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.568769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44461 (local address 127.0.98.60:33633)
0929 16:26:27.568928 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.568933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.569915 (+ 982us) server_negotiation.cc:407] Connection header received
0929 16:26:27.571121 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.571125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.571178 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.571231 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:27.572650 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.573479 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.574926 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.575284 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.575711 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.579929 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:27.579933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:27.579936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:27.579967 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:27.581864 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.582535 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.582540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.582545 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.582646 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.583075 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.583081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.583084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.583315 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:27.583408 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.585398 (+ 1990us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.585532 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:26:27.585695 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.569250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36225 (local address 127.0.98.3:43773)
0929 16:26:27.570487 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:27.570510 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:27.570899 (+ 389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:27.571439 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:27.571447 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:27.571917 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:27.572491 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.572505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.573376 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.573379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.574760 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.574769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.575527 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.575533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.575659 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.576227 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:27.576242 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:27.579318 (+ 3076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:27.582413 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:27.582416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:27.582419 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:27.582837 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:27.583147 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:27.583150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:27.583152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:27.583254 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:27.583671 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:27.583674 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:27.583886 (+ 212us) client_negotiation.cc:769] Sending connection context
0929 16:26:27.584117 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 16:26:27.585284 (+ 1167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1134,"mutex_wait_us":959,"spinlock_wait_cycles":114304,"thread_start_us":175,"threads_started":1}
I20260929 16:26:27.585978 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.569248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33633 (local address 127.0.98.3:44461)
0929 16:26:27.569756 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:27.569818 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:27.570892 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:27.571441 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:27.571448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:27.571989 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:27.572506 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.572512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.573590 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.573595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.574759 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.574769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.575880 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.575884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.575965 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.576403 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:27.576420 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:27.579774 (+ 3354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:27.582000 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:27.582007 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:27.582028 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:27.582411 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:27.582755 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:27.582758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:27.582760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:27.582970 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:27.583470 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:27.583478 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:27.583889 (+ 411us) client_negotiation.cc:769] Sending connection context
0929 16:26:27.584119 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 16:26:27.584505 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":226,"spinlock_wait_cycles":12800,"thread_start_us":131,"threads_started":1}
I20260929 16:26:27.586035 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.569248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.3:51863)
0929 16:26:27.570309 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:27.570324 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:27.570892 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:27.571441 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:27.571448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:27.571989 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:27.572503 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.572512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.573621 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.573624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.574380 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.574389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.574886 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.574890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.575076 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.575718 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:27.575736 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:27.580228 (+ 4492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:27.582413 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:27.582416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:27.582419 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:27.582837 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:27.583151 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:27.583153 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:27.583154 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:27.583259 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:27.584006 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:27.584009 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:27.584103 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 16:26:27.584749 (+ 646us) client_negotiation.cc:240] Negotiation successful
0929 16:26:27.584895 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"mutex_wait_us":82,"spinlock_wait_cycles":134784,"thread_start_us":176,"threads_started":1}
I20260929 16:26:27.587265 7760 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44555
I20260929 16:26:27.587265 7759 heartbeater.cc:347] Connected to a master server at 127.0.98.60:33633
I20260929 16:26:27.587558 7759 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:27.588195 7759 heartbeater.cc:511] Master 127.0.98.60:33633 requested a full tablet report, sending...
I20260929 16:26:27.587581 7761 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36225
I20260929 16:26:27.588840 7761 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:27.589015 7761 heartbeater.cc:511] Master 127.0.98.61:36225 requested a full tablet report, sending...
I20260929 16:26:27.587558 7760 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:27.589399 7760 heartbeater.cc:511] Master 127.0.98.62:44555 requested a full tablet report, sending...
I20260929 16:26:27.589963 7124 ts_manager.cc:194] Registered new tserver with Master: 4bcbbb5789854afbbea071a979eccfb7 (127.0.98.3:46539)
I20260929 16:26:27.590086 7306 ts_manager.cc:194] Registered new tserver with Master: 4bcbbb5789854afbbea071a979eccfb7 (127.0.98.3:46539)
I20260929 16:26:27.590521 6516 ts_manager.cc:194] Re-registered known tserver with Master: 4bcbbb5789854afbbea071a979eccfb7 (127.0.98.3:46539)
I20260929 16:26:27.591137 6516 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51863
I20260929 16:26:27.596691 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.579190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59486 (local address 127.0.98.61:36225)
0929 16:26:27.584793 (+ 5603us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.584796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.584808 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:26:27.584840 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.584843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.584889 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.584968 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:27.587609 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.588354 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.590666 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.590987 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.591065 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.592932 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:27.592939 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:27.592945 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:27.592970 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:27.594667 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.595306 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.595311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.595315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.595401 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.595710 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.595713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.595715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.595967 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:27.596109 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.596384 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.596496 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5542,"thread_start_us":74,"threads_started":1}
I20260929 16:26:27.597026 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.579417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43258 (local address 127.0.98.62:44555)
0929 16:26:27.579880 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.579885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.580216 (+ 331us) server_negotiation.cc:407] Connection header received
0929 16:26:27.580282 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.580286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.580346 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.580441 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:27.581637 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.582459 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.588321 (+ 5862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.588671 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.588785 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.590875 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:27.590879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:27.590882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:27.590908 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:27.594712 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.595355 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.595359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.595361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.595439 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.595768 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.595771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.595772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.595975 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:27.596089 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.596568 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.596714 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":72,"threads_started":1}
I20260929 16:26:27.597083 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.579551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43938 (local address 127.0.98.60:33633)
0929 16:26:27.580463 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.580468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.582939 (+ 2471us) server_negotiation.cc:407] Connection header received
0929 16:26:27.582976 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.582979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.583032 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.583111 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:27.584041 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.587777 (+ 3736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.588942 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.589359 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.589465 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.592976 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:27.592980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:27.592982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:27.593008 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:27.594765 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.595345 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.595348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.595350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.595439 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.595751 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.595756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.595757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.595971 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:27.596086 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.596795 (+ 709us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.596934 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":87,"threads_started":1}
I20260929 16:26:27.600792 6516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:26:27.603969 6516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:26:27.607152 7078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d732cddd2a8490ea5a12bdcc1798761" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: true } } }
I20260929 16:26:27.607316 7078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:27.611792 7777 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761: attempting to promote NON_VOTER fea1cb04b3954e36952f2144017d4d8c to VOTER
I20260929 16:26:27.612342 7777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: false } }
I20260929 16:26:27.614187 7134 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 1 FOLLOWER]: Refusing update from remote peer 6d732cddd2a8490ea5a12bdcc1798761: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 16:26:27.614262 7311 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 1 LEARNER]: Refusing update from remote peer 6d732cddd2a8490ea5a12bdcc1798761: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 16:26:27.614581 7778 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 16:26:27.614887 7778 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 16:26:27.616616 7078 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fea1cb04b3954e36952f2144017d4d8c (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: false } } }
I20260929 16:26:27.617667 7134 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fea1cb04b3954e36952f2144017d4d8c (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: false } } }
I20260929 16:26:27.618203 7078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6d732cddd2a8490ea5a12bdcc1798761" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: false } } }
I20260929 16:26:27.618304 7078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:27.617959 7311 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fea1cb04b3954e36952f2144017d4d8c (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: false } } }
I20260929 16:26:27.618451 7078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d732cddd2a8490ea5a12bdcc1798761" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: false } } }
I20260929 16:26:27.618537 7078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:27.618685 7078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d732cddd2a8490ea5a12bdcc1798761" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: false } } }
I20260929 16:26:27.618779 7078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:27.619808 7786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d732cddd2a8490ea5a12bdcc1798761" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: false } } }
I20260929 16:26:27.619956 7786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:26:27.621292 7349 sys_catalog.cc:455] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d732cddd2a8490ea5a12bdcc1798761" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: false } } }
I20260929 16:26:27.621412 7349 sys_catalog.cc:458] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:26:27.626133 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.620692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46539 (local address 127.0.0.1:36380)
0929 16:26:27.620792 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:27.620807 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:27.620895 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:27.621776 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:27.621779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:27.621800 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:27.621913 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.621919 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.623350 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.623356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.624475 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.624482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.625732 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.625735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.625844 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.625865 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:26:27.625925 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 16:26:27.625998 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":1}
I20260929 16:26:27.626202 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.620854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36380 (local address 127.0.98.3:46539)
0929 16:26:27.621232 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.621247 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.621273 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:26:27.621357 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.621363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.621607 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.622076 (+ 469us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:27.622219 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.623245 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.624595 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.625605 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.625734 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.625897 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.625963 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.626054 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":132,"threads_started":1}
I20260929 16:26:27.627291 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.620955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41403 (local address 127.0.0.1:41382)
0929 16:26:27.621942 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:27.621958 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:27.622037 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:27.622651 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:27.622654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:27.622670 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:27.622761 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.622766 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.623958 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.623962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.625620 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.625632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.626880 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.626887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.626969 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.626991 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:26:27.627070 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 16:26:27.627154 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"thread_start_us":96,"threads_started":1}
I20260929 16:26:27.627552 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.621371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41382 (local address 127.0.98.1:41403)
0929 16:26:27.621793 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.621807 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.622120 (+ 313us) server_negotiation.cc:407] Connection header received
0929 16:26:27.622200 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.622205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.622385 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.622524 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:27.622950 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.623815 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.625750 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.626586 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.626691 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.626820 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.627123 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.627229 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":62,"threads_started":1}
I20260929 16:26:27.628263 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.622076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43549 (local address 127.0.0.1:33318)
0929 16:26:27.622289 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:27.622303 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:27.622392 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:27.624109 (+ 1717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:27.624112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:27.624127 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:27.624216 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.624221 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.625641 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.625643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.626789 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.626799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.627888 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.627891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.627973 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.627993 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 16:26:27.628056 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 16:26:27.628134 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":26,"thread_start_us":98,"threads_started":1}
I20260929 16:26:27.628459 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.621785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33318 (local address 127.0.98.2:43549)
0929 16:26:27.623534 (+ 1749us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.623545 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.623567 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:26:27.623635 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.623640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.623829 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.623978 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:27.624346 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.625498 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.626930 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.627749 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.627889 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.628022 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.628078 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.628163 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1653,"thread_start_us":96,"threads_started":1}
I20260929 16:26:27.629565 7693 tablet_service.cc:1467] Processing CreateTablet for tablet 07a1b768c2ce4e0ab5caac416cfcb9b2 (DEFAULT_TABLE table=test-table [id=81e5a540755c46918129737f66c72a71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:27.630790 7693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a1b768c2ce4e0ab5caac416cfcb9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:27.630524 7420 tablet_service.cc:1467] Processing CreateTablet for tablet 07a1b768c2ce4e0ab5caac416cfcb9b2 (DEFAULT_TABLE table=test-table [id=81e5a540755c46918129737f66c72a71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:27.631528 7420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a1b768c2ce4e0ab5caac416cfcb9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:27.631172 7557 tablet_service.cc:1467] Processing CreateTablet for tablet 07a1b768c2ce4e0ab5caac416cfcb9b2 (DEFAULT_TABLE table=test-table [id=81e5a540755c46918129737f66c72a71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:27.632027 7557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a1b768c2ce4e0ab5caac416cfcb9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:27.638821 7792 tablet_bootstrap.cc:492] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6: Bootstrap starting.
I20260929 16:26:27.640084 7794 tablet_bootstrap.cc:492] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 13adcbd1e74248ab871d13068c0f23c9: Bootstrap starting.
I20260929 16:26:27.641530 7793 tablet_bootstrap.cc:492] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 4bcbbb5789854afbbea071a979eccfb7: Bootstrap starting.
I20260929 16:26:27.641644 7792 tablet_bootstrap.cc:654] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:27.642299 7794 tablet_bootstrap.cc:654] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 13adcbd1e74248ab871d13068c0f23c9: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:27.642421 7792 log.cc:824] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:27.643160 7794 log.cc:824] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 13adcbd1e74248ab871d13068c0f23c9: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:27.643818 7793 tablet_bootstrap.cc:654] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 4bcbbb5789854afbbea071a979eccfb7: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:27.644508 7792 tablet_bootstrap.cc:492] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6: No bootstrap required, opened a new log
I20260929 16:26:27.644647 7793 log.cc:824] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 4bcbbb5789854afbbea071a979eccfb7: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:27.644726 7792 ts_tablet_manager.cc:1397] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:26:27.645186 7794 tablet_bootstrap.cc:492] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 13adcbd1e74248ab871d13068c0f23c9: No bootstrap required, opened a new log
I20260929 16:26:27.645416 7794 ts_tablet_manager.cc:1397] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 13adcbd1e74248ab871d13068c0f23c9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260929 16:26:27.646656 7793 tablet_bootstrap.cc:492] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 4bcbbb5789854afbbea071a979eccfb7: No bootstrap required, opened a new log
I20260929 16:26:27.646854 7793 ts_tablet_manager.cc:1397] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 4bcbbb5789854afbbea071a979eccfb7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:26:27.649919 7792 raft_consensus.cc:348] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcbbb5789854afbbea071a979eccfb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46539 } } peers { permanent_uuid: "13adcbd1e74248ab871d13068c0f23c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41403 } } peers { permanent_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43549 } }
I20260929 16:26:27.650230 7792 raft_consensus.cc:374] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:27.650295 7792 raft_consensus.cc:729] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcbdbf5a43df466cb16a8911c91e24a6, State: Initialized, Role: FOLLOWER
I20260929 16:26:27.650209 7794 raft_consensus.cc:348] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 13adcbd1e74248ab871d13068c0f23c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcbbb5789854afbbea071a979eccfb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46539 } } peers { permanent_uuid: "13adcbd1e74248ab871d13068c0f23c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41403 } } peers { permanent_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43549 } }
I20260929 16:26:27.650512 7794 raft_consensus.cc:374] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 13adcbd1e74248ab871d13068c0f23c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:27.650574 7794 raft_consensus.cc:729] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 13adcbd1e74248ab871d13068c0f23c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13adcbd1e74248ab871d13068c0f23c9, State: Initialized, Role: FOLLOWER
I20260929 16:26:27.650780 7792 consensus_queue.cc:260] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcbbb5789854afbbea071a979eccfb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46539 } } peers { permanent_uuid: "13adcbd1e74248ab871d13068c0f23c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41403 } } peers { permanent_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43549 } }
I20260929 16:26:27.651033 7794 consensus_queue.cc:260] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 13adcbd1e74248ab871d13068c0f23c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcbbb5789854afbbea071a979eccfb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46539 } } peers { permanent_uuid: "13adcbd1e74248ab871d13068c0f23c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41403 } } peers { permanent_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43549 } }
I20260929 16:26:27.651562 7793 raft_consensus.cc:348] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 4bcbbb5789854afbbea071a979eccfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcbbb5789854afbbea071a979eccfb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46539 } } peers { permanent_uuid: "13adcbd1e74248ab871d13068c0f23c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41403 } } peers { permanent_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43549 } }
I20260929 16:26:27.651819 7793 raft_consensus.cc:374] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 4bcbbb5789854afbbea071a979eccfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:27.651860 7792 ts_tablet_manager.cc:1428] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 16:26:27.651907 7793 raft_consensus.cc:729] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 4bcbbb5789854afbbea071a979eccfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcbbb5789854afbbea071a979eccfb7, State: Initialized, Role: FOLLOWER
I20260929 16:26:27.652058 7794 ts_tablet_manager.cc:1428] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 13adcbd1e74248ab871d13068c0f23c9: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260929 16:26:27.652320 7793 consensus_queue.cc:260] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 4bcbbb5789854afbbea071a979eccfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcbbb5789854afbbea071a979eccfb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46539 } } peers { permanent_uuid: "13adcbd1e74248ab871d13068c0f23c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41403 } } peers { permanent_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43549 } }
I20260929 16:26:27.653254 7624 heartbeater.cc:503] Master 127.0.98.62:44555 was elected leader, sending a full tablet report...
I20260929 16:26:27.653275 7793 ts_tablet_manager.cc:1428] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 4bcbbb5789854afbbea071a979eccfb7: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260929 16:26:27.654614 7760 heartbeater.cc:503] Master 127.0.98.62:44555 was elected leader, sending a full tablet report...
I20260929 16:26:27.655961 7487 heartbeater.cc:503] Master 127.0.98.62:44555 was elected leader, sending a full tablet report...
I20260929 16:26:27.693172 7798 raft_consensus.cc:482] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:26:27.693447 7798 raft_consensus.cc:504] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcbbb5789854afbbea071a979eccfb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46539 } } peers { permanent_uuid: "13adcbd1e74248ab871d13068c0f23c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41403 } } peers { permanent_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43549 } }
I20260929 16:26:27.694607 7798 leader_election.cc:288] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bcbbb5789854afbbea071a979eccfb7 (127.0.98.3:46539), 13adcbd1e74248ab871d13068c0f23c9 (127.0.98.1:41403)
I20260929 16:26:27.698964 7629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.694795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46539 (local address 127.0.98.2:55279)
0929 16:26:27.694933 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:27.694951 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:27.695049 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:27.695587 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:27.695589 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:27.695603 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:27.695723 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.695730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.696571 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.696574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.697843 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.697853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.698503 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.698508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.698614 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.698645 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:26:27.698753 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 16:26:27.698831 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":78}
I20260929 16:26:27.698963 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.695150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55279 (local address 127.0.98.3:46539)
0929 16:26:27.695301 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.695305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.695317 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:26:27.695359 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.695362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.695414 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.695482 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:27.695873 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.696475 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.697986 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.698380 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.698475 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.698558 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.698770 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.698844 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:26:27.699723 7713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a1b768c2ce4e0ab5caac416cfcb9b2" candidate_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcbbb5789854afbbea071a979eccfb7" is_pre_election: true
I20260929 16:26:27.700024 7627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.694926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41403 (local address 127.0.98.2:39913)
0929 16:26:27.695030 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:27.695063 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:27.695149 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:27.695487 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:27.695492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:27.695516 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:27.695614 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.695619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.696417 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.696422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.697843 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:27.697853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.699618 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.699622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:27.699728 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.699759 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:26:27.699839 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 16:26:27.699912 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":52}
I20260929 16:26:27.700076 7713 raft_consensus.cc:2454] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 4bcbbb5789854afbbea071a979eccfb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcbdbf5a43df466cb16a8911c91e24a6 in term 0.
I20260929 16:26:27.700117 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.695106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39913 (local address 127.0.98.1:41403)
0929 16:26:27.695244 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.695249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.695264 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:26:27.695319 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.695321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.695372 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.695453 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:27.695727 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.696311 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.698624 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.699185 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.699295 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.699380 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.699908 (+ 528us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.700001 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:26:27.700675 7513 leader_election.cc:302] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcbbb5789854afbbea071a979eccfb7, bcbdbf5a43df466cb16a8911c91e24a6; no voters:
I20260929 16:26:27.700776 7440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a1b768c2ce4e0ab5caac416cfcb9b2" candidate_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13adcbd1e74248ab871d13068c0f23c9" is_pre_election: true
I20260929 16:26:27.701013 7798 raft_consensus.cc:2781] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:26:27.701110 7798 raft_consensus.cc:482] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:26:27.701062 7440 raft_consensus.cc:2454] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 13adcbd1e74248ab871d13068c0f23c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcbdbf5a43df466cb16a8911c91e24a6 in term 0.
I20260929 16:26:27.701175 7798 raft_consensus.cc:3037] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:27.702380 7798 raft_consensus.cc:504] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcbbb5789854afbbea071a979eccfb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46539 } } peers { permanent_uuid: "13adcbd1e74248ab871d13068c0f23c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41403 } } peers { permanent_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43549 } }
I20260929 16:26:27.702723 7798 leader_election.cc:288] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6 [CANDIDATE]: Term 1 election: Requested vote from peers 4bcbbb5789854afbbea071a979eccfb7 (127.0.98.3:46539), 13adcbd1e74248ab871d13068c0f23c9 (127.0.98.1:41403)
I20260929 16:26:27.703082 7440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a1b768c2ce4e0ab5caac416cfcb9b2" candidate_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13adcbd1e74248ab871d13068c0f23c9"
I20260929 16:26:27.703086 7713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a1b768c2ce4e0ab5caac416cfcb9b2" candidate_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcbbb5789854afbbea071a979eccfb7"
I20260929 16:26:27.703226 7440 raft_consensus.cc:3037] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 13adcbd1e74248ab871d13068c0f23c9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:27.703303 7713 raft_consensus.cc:3037] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 4bcbbb5789854afbbea071a979eccfb7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:27.704494 7440 raft_consensus.cc:2454] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 13adcbd1e74248ab871d13068c0f23c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcbdbf5a43df466cb16a8911c91e24a6 in term 1.
I20260929 16:26:27.704494 7713 raft_consensus.cc:2454] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 4bcbbb5789854afbbea071a979eccfb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcbdbf5a43df466cb16a8911c91e24a6 in term 1.
I20260929 16:26:27.704818 7512 leader_election.cc:302] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13adcbd1e74248ab871d13068c0f23c9, bcbdbf5a43df466cb16a8911c91e24a6; no voters:
I20260929 16:26:27.704998 7798 raft_consensus.cc:2781] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:27.705265 7798 raft_consensus.cc:686] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6 [term 1 LEADER]: Becoming Leader. State: Replica: bcbdbf5a43df466cb16a8911c91e24a6, State: Running, Role: LEADER
I20260929 16:26:27.705605 7798 consensus_queue.cc:237] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcbbb5789854afbbea071a979eccfb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46539 } } peers { permanent_uuid: "13adcbd1e74248ab871d13068c0f23c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41403 } } peers { permanent_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43549 } }
I20260929 16:26:27.708950 6516 catalog_manager.cc:5495] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcbdbf5a43df466cb16a8911c91e24a6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcbbb5789854afbbea071a979eccfb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13adcbd1e74248ab871d13068c0f23c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43549 } health_report { overall_health: HEALTHY } } }
I20260929 16:26:27.727815 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.723908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33330 (local address 127.0.98.2:43549)
0929 16:26:27.724059 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.724064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.724080 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:26:27.724354 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.724357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.724418 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.724536 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:27.724792 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.725433 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.726295 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.726520 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.726643 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.726954 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:27.727397 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:27.727538 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.727585 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.727657 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:26:27.734378 7440 raft_consensus.cc:1261] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 13adcbd1e74248ab871d13068c0f23c9 [term 1 FOLLOWER]: Refusing update from remote peer bcbdbf5a43df466cb16a8911c91e24a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:27.734378 7713 raft_consensus.cc:1261] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 4bcbbb5789854afbbea071a979eccfb7 [term 1 FOLLOWER]: Refusing update from remote peer bcbdbf5a43df466cb16a8911c91e24a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:27.735247 7801 consensus_queue.cc:1035] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13adcbd1e74248ab871d13068c0f23c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:27.735452 7798 consensus_queue.cc:1035] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcbbb5789854afbbea071a979eccfb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:27.750106 7806 mvcc.cc:205] Tried to move back new op lower bound from 7334703872951820288 to 7334703872844537856. Current Snapshot: MvccSnapshot[applied={T|T < 7334703872951820288}]
I20260929 16:26:27.754941 6515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:27.755160 6515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:27.760286 6515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d732cddd2a8490ea5a12bdcc1798761: Sending DeleteTablet for 3 replicas of tablet 07a1b768c2ce4e0ab5caac416cfcb9b2
I20260929 16:26:27.761337 7693 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a1b768c2ce4e0ab5caac416cfcb9b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36380
I20260929 16:26:27.762339 7557 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a1b768c2ce4e0ab5caac416cfcb9b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33318
I20260929 16:26:27.762748 7420 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a1b768c2ce4e0ab5caac416cfcb9b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:27 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41382
I20260929 16:26:27.763010 7817 tablet_replica.cc:330] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6: stopping tablet replica
I20260929 16:26:27.763279 7818 tablet_replica.cc:330] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 13adcbd1e74248ab871d13068c0f23c9: stopping tablet replica
I20260929 16:26:27.763540 7817 raft_consensus.cc:2229] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:26:27.763597 7818 raft_consensus.cc:2229] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 13adcbd1e74248ab871d13068c0f23c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:27.763960 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 6484
I20260929 16:26:27.763981 7817 raft_consensus.cc:2258] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:27.764106 7818 raft_consensus.cc:2258] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 13adcbd1e74248ab871d13068c0f23c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:27.766232 7817 ts_tablet_manager.cc:1905] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:27.766722 7818 ts_tablet_manager.cc:1905] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 13adcbd1e74248ab871d13068c0f23c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:27.769299 7817 ts_tablet_manager.cc:1918] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:26:27.769410 7817 log.cc:1196] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/07a1b768c2ce4e0ab5caac416cfcb9b2
I20260929 16:26:27.769662 7818 ts_tablet_manager.cc:1918] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 13adcbd1e74248ab871d13068c0f23c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:26:27.769776 7817 ts_tablet_manager.cc:1939] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P bcbdbf5a43df466cb16a8911c91e24a6: Deleting consensus metadata
I20260929 16:26:27.769876 7818 log.cc:1196] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 13adcbd1e74248ab871d13068c0f23c9: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-0/wal/wals/07a1b768c2ce4e0ab5caac416cfcb9b2
I20260929 16:26:27.770263 7818 ts_tablet_manager.cc:1939] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 13adcbd1e74248ab871d13068c0f23c9: Deleting consensus metadata
W20260929 16:26:27.771719 7510 connection.cc:535] server connection from 127.0.0.1:33318 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 16:26:27.771721 7375 connection.cc:535] server connection from 127.0.0.1:41382 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 16:26:27.773430 7815 tablet_replica.cc:330] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 4bcbbb5789854afbbea071a979eccfb7: stopping tablet replica
I20260929 16:26:27.773851 7815 raft_consensus.cc:2229] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 4bcbbb5789854afbbea071a979eccfb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:27.774255 7815 raft_consensus.cc:2258] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 4bcbbb5789854afbbea071a979eccfb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:27.776145 7815 ts_tablet_manager.cc:1905] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 4bcbbb5789854afbbea071a979eccfb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:27.779258 7815 ts_tablet_manager.cc:1918] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 4bcbbb5789854afbbea071a979eccfb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:26:27.779407 7815 log.cc:1196] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 4bcbbb5789854afbbea071a979eccfb7: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/07a1b768c2ce4e0ab5caac416cfcb9b2
I20260929 16:26:27.779767 7815 ts_tablet_manager.cc:1939] T 07a1b768c2ce4e0ab5caac416cfcb9b2 P 4bcbbb5789854afbbea071a979eccfb7: Deleting consensus metadata
W20260929 16:26:27.780946 7648 connection.cc:441] server connection from 127.0.0.1:36380 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36380 (request call id 1) could send its response
I20260929 16:26:27.783097 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.763467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59496 (local address 127.0.98.61:36225)
0929 16:26:27.763572 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.763577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.773162 (+ 9585us) server_negotiation.cc:407] Connection header received
0929 16:26:27.773379 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.773382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.773434 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.773486 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:27.774451 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.775263 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.776450 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.776814 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.776885 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.779245 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:27.779249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:27.779252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:27.779281 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:27.781262 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.781729 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.781732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.781734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.781817 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.782161 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.782172 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.782174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.782378 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:27.782518 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.782772 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.782892 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 16:26:27.785604 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.775181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43956 (local address 127.0.98.60:33633)
0929 16:26:27.775298 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.775305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.776740 (+ 1435us) server_negotiation.cc:407] Connection header received
0929 16:26:27.776915 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.776918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.776965 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.777016 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:27.778063 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.778859 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.780061 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.780358 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.780434 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.782107 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:27.782118 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:27.782121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:27.782147 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:27.783865 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.784335 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.784338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.784340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.784424 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.784678 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.784683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.784687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.784859 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:27.784998 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.785315 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.785428 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 16:26:27.791766 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.763336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43942 (local address 127.0.98.60:33633)
0929 16:26:27.763458 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.763463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.783104 (+ 19641us) server_negotiation.cc:407] Connection header received
0929 16:26:27.783280 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.783283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.783326 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.783374 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:27.784077 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.784909 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.786023 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.786423 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.786518 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.788178 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:27.788182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:27.788184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:27.788208 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:27.790093 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.790540 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.790545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.790547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.790630 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.790898 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.790901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.790903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.791126 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:27.791261 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.791487 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.791617 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 16:26:27.793658 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:27.776237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59512 (local address 127.0.98.61:36225)
0929 16:26:27.776360 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:27.776366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:27.785662 (+ 9296us) server_negotiation.cc:407] Connection header received
0929 16:26:27.785813 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:27.785816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:27.785860 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:27.785915 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:27.786915 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.787600 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.788378 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:27.788770 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:27.788889 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:27.790101 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:27.790105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:27.790108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:27.790130 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:27.791809 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.792380 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.792383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.792385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.792465 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:27.792756 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:27.792759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:27.792760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:27.792929 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:27.793029 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:27.793442 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 16:26:27.793539 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 16:26:28.749128 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:28.748579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.1:33265)
0929 16:26:28.748882 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:28.748986 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":144,"threads_started":1}
W20260929 16:26:28.749580 7487 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44555 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
I20260929 16:26:28.750605 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:28.750131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.2:47079)
0929 16:26:28.750393 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:28.750487 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":83,"threads_started":1}
I20260929 16:26:28.751003 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:28.750658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.3:40305)
0929 16:26:28.750865 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:28.750930 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":61,"threads_started":1}
W20260929 16:26:28.751032 7624 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44555 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
W20260929 16:26:28.751345 7760 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44555 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
I20260929 16:26:29.260897 7829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d732cddd2a8490ea5a12bdcc1798761)
I20260929 16:26:29.261165 7829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: false } }
I20260929 16:26:29.261493 7830 raft_consensus.cc:482] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d732cddd2a8490ea5a12bdcc1798761)
I20260929 16:26:29.261622 7830 raft_consensus.cc:504] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: false } }
I20260929 16:26:29.262171 7830 leader_election.cc:288] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d732cddd2a8490ea5a12bdcc1798761 (127.0.98.62:44555), 7bc65de05f0c4d39a89442538492b338 (127.0.98.61:36225)
I20260929 16:26:29.262804 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:29.262333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.0.1:43748)
0929 16:26:29.262603 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:29.262696 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":73,"threads_started":1}
W20260929 16:26:29.263082 7292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
I20260929 16:26:29.263124 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:29.262453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.0.1:43754)
0929 16:26:29.262963 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:29.263034 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":68,"threads_started":1}
W20260929 16:26:29.263247 7109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
I20260929 16:26:29.263324 7829 leader_election.cc:288] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d732cddd2a8490ea5a12bdcc1798761 (127.0.98.62:44555), fea1cb04b3954e36952f2144017d4d8c (127.0.98.60:33633)
I20260929 16:26:29.264165 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:29.263907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.0.1:43768)
0929 16:26:29.264036 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:29.264090 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 16:26:29.264225 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:29.263969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.0.1:43782)
0929 16:26:29.264106 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:29.264153 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260929 16:26:29.264307 7109 leader_election.cc:334] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d732cddd2a8490ea5a12bdcc1798761 (127.0.98.62:44555): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
W20260929 16:26:29.264379 7292 leader_election.cc:334] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d732cddd2a8490ea5a12bdcc1798761 (127.0.98.62:44555): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
I20260929 16:26:29.266976 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:29.262455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36225 (local address 127.0.0.1:59514)
0929 16:26:29.262603 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:29.262620 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:29.262760 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:29.263319 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:29.263322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:29.263345 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:29.263458 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:29.263465 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:29.264482 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:29.264485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:29.265667 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:29.265677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:29.266629 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:29.266632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:29.266703 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:29.266723 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 16:26:29.266806 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 16:26:29.266868 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260929 16:26:29.267042 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:29.262684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59514 (local address 127.0.98.61:36225)
0929 16:26:29.262944 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:29.262951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:29.262968 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:26:29.263017 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:29.263023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:29.263080 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:29.263170 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:29.263590 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:29.264356 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:29.265808 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:29.266488 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:29.266605 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:29.266667 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:29.266844 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 16:26:29.266915 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":187,"thread_start_us":83,"threads_started":1}
I20260929 16:26:29.267614 7134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fea1cb04b3954e36952f2144017d4d8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7bc65de05f0c4d39a89442538492b338" is_pre_election: true
I20260929 16:26:29.267776 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:29.262898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33633 (local address 127.0.0.1:43968)
0929 16:26:29.263501 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:29.263516 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:29.263591 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:29.263922 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:29.263926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:29.263944 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:29.264036 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:29.264044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:29.265004 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:29.265008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:29.266316 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:29.266326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:29.267383 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:29.267386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:29.267449 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:29.267466 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 16:26:29.267567 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 16:26:29.267660 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":96,"thread_start_us":46,"threads_started":1}
I20260929 16:26:29.267827 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:29.263203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43968 (local address 127.0.98.60:33633)
0929 16:26:29.263659 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:29.263664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:29.263678 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:26:29.263719 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:29.263722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:29.263793 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:29.264142 (+ 349us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:29.264239 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:29.264883 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:29.266447 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:29.267077 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:29.267258 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:29.267331 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:29.267600 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 16:26:29.267668 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":61,"threads_started":1}
I20260929 16:26:29.267776 7134 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fea1cb04b3954e36952f2144017d4d8c in term 1.
I20260929 16:26:29.268173 7291 leader_election.cc:302] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7bc65de05f0c4d39a89442538492b338, fea1cb04b3954e36952f2144017d4d8c; no voters: 6d732cddd2a8490ea5a12bdcc1798761
I20260929 16:26:29.268222 7311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7bc65de05f0c4d39a89442538492b338" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fea1cb04b3954e36952f2144017d4d8c" is_pre_election: true
I20260929 16:26:29.268422 7830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 16:26:29.268543 7311 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bc65de05f0c4d39a89442538492b338 in term 1.
I20260929 16:26:29.268649 7830 raft_consensus.cc:482] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d732cddd2a8490ea5a12bdcc1798761)
I20260929 16:26:29.268719 7830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:26:29.268906 7110 leader_election.cc:302] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7bc65de05f0c4d39a89442538492b338, fea1cb04b3954e36952f2144017d4d8c; no voters: 6d732cddd2a8490ea5a12bdcc1798761
I20260929 16:26:29.269155 7829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 16:26:29.269248 7829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d732cddd2a8490ea5a12bdcc1798761)
I20260929 16:26:29.269295 7829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:26:29.270254 7830 raft_consensus.cc:504] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: false } }
I20260929 16:26:29.270522 7829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: false } }
I20260929 16:26:29.270676 7830 leader_election.cc:288] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [CANDIDATE]: Term 2 election: Requested vote from peers 6d732cddd2a8490ea5a12bdcc1798761 (127.0.98.62:44555), 7bc65de05f0c4d39a89442538492b338 (127.0.98.61:36225)
I20260929 16:26:29.270947 7829 leader_election.cc:288] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [CANDIDATE]: Term 2 election: Requested vote from peers 6d732cddd2a8490ea5a12bdcc1798761 (127.0.98.62:44555), fea1cb04b3954e36952f2144017d4d8c (127.0.98.60:33633)
I20260929 16:26:29.271248 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:29.271035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.0.1:43808)
0929 16:26:29.271126 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:29.271188 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 16:26:29.271390 7311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7bc65de05f0c4d39a89442538492b338" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fea1cb04b3954e36952f2144017d4d8c"
I20260929 16:26:29.271497 7134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fea1cb04b3954e36952f2144017d4d8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7bc65de05f0c4d39a89442538492b338"
I20260929 16:26:29.271579 7311 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7bc65de05f0c4d39a89442538492b338 in current term 2: Already voted for candidate fea1cb04b3954e36952f2144017d4d8c in this term.
I20260929 16:26:29.271638 7134 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fea1cb04b3954e36952f2144017d4d8c in current term 2: Already voted for candidate 7bc65de05f0c4d39a89442538492b338 in this term.
I20260929 16:26:29.271941 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:29.271673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.0.1:43824)
0929 16:26:29.271812 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:29.271879 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 16:26:29.271375 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:29.270890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.0.1:43796)
0929 16:26:29.271178 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:29.271234 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
W20260929 16:26:29.272075 7109 leader_election.cc:334] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6d732cddd2a8490ea5a12bdcc1798761 (127.0.98.62:44555): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
I20260929 16:26:29.272148 7109 leader_election.cc:302] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7bc65de05f0c4d39a89442538492b338; no voters: 6d732cddd2a8490ea5a12bdcc1798761, fea1cb04b3954e36952f2144017d4d8c
I20260929 16:26:29.272301 7829 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 16:26:29.272729 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:29.272465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.0.1:43828)
0929 16:26:29.272629 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:29.272681 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 16:26:29.272871 7292 leader_election.cc:334] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6d732cddd2a8490ea5a12bdcc1798761 (127.0.98.62:44555): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
I20260929 16:26:29.272946 7292 leader_election.cc:302] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fea1cb04b3954e36952f2144017d4d8c; no voters: 6d732cddd2a8490ea5a12bdcc1798761, 7bc65de05f0c4d39a89442538492b338
I20260929 16:26:29.273073 7830 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 16:26:29.750893 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:29.750235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.1:49303)
0929 16:26:29.750702 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:29.750778 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":102,"threads_started":1}
I20260929 16:26:29.752066 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:29.751569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.2:39947)
0929 16:26:29.751906 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:29.751973 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":92,"threads_started":1}
I20260929 16:26:29.752274 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:29.751825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.3:60031)
0929 16:26:29.752090 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:29.752153 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":93,"threads_started":1}
I20260929 16:26:30.753325 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:30.752667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.1:53761)
0929 16:26:30.753113 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:30.753182 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":82,"threads_started":1}
I20260929 16:26:30.753366 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:30.752856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.2:42935)
0929 16:26:30.753186 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:30.753278 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":71,"threads_started":1}
I20260929 16:26:30.753562 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:30.753151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.3:58487)
0929 16:26:30.753386 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:30.753453 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":95,"threads_started":1}
I20260929 16:26:30.871209 7845 raft_consensus.cc:482] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:26:30.871377 7845 raft_consensus.cc:504] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: false } }
I20260929 16:26:30.871819 7845 leader_election.cc:288] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d732cddd2a8490ea5a12bdcc1798761 (127.0.98.62:44555), 7bc65de05f0c4d39a89442538492b338 (127.0.98.61:36225)
I20260929 16:26:30.872324 7134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fea1cb04b3954e36952f2144017d4d8c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7bc65de05f0c4d39a89442538492b338" is_pre_election: true
I20260929 16:26:30.872509 7134 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fea1cb04b3954e36952f2144017d4d8c in term 2.
I20260929 16:26:30.872547 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:30.872127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.0.1:44062)
0929 16:26:30.872356 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:30.872425 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"spinlock_wait_cycles":3456,"thread_start_us":135,"threads_started":1}
I20260929 16:26:30.872844 7291 leader_election.cc:302] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bc65de05f0c4d39a89442538492b338, fea1cb04b3954e36952f2144017d4d8c; no voters:
I20260929 16:26:30.873001 7845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 16:26:30.873070 7845 raft_consensus.cc:482] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:26:30.873128 7845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 2 FOLLOWER]: Advancing to term 3
I20260929 16:26:30.873432 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:30.873195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.0.1:44078)
0929 16:26:30.873315 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:30.873367 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 16:26:30.873529 7292 leader_election.cc:334] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d732cddd2a8490ea5a12bdcc1798761 (127.0.98.62:44555): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
I20260929 16:26:30.874266 7845 raft_consensus.cc:504] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: false } }
I20260929 16:26:30.874606 7845 leader_election.cc:288] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [CANDIDATE]: Term 3 election: Requested vote from peers 6d732cddd2a8490ea5a12bdcc1798761 (127.0.98.62:44555), 7bc65de05f0c4d39a89442538492b338 (127.0.98.61:36225)
I20260929 16:26:30.874933 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:30.874708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.0.1:44086)
0929 16:26:30.874829 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:30.874877 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 16:26:30.874938 7134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fea1cb04b3954e36952f2144017d4d8c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7bc65de05f0c4d39a89442538492b338"
I20260929 16:26:30.875074 7134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 2 FOLLOWER]: Advancing to term 3
I20260929 16:26:30.875501 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:30.875307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.0.1:44090)
0929 16:26:30.875406 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:30.875448 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 16:26:30.875591 7292 leader_election.cc:334] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6d732cddd2a8490ea5a12bdcc1798761 (127.0.98.62:44555): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
I20260929 16:26:30.876082 7134 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fea1cb04b3954e36952f2144017d4d8c in term 3.
I20260929 16:26:30.876381 7291 leader_election.cc:302] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7bc65de05f0c4d39a89442538492b338, fea1cb04b3954e36952f2144017d4d8c; no voters: 6d732cddd2a8490ea5a12bdcc1798761
I20260929 16:26:30.876601 7845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 3 FOLLOWER]: Leader election won for term 3
I20260929 16:26:30.876791 7845 raft_consensus.cc:686] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [term 3 LEADER]: Becoming Leader. State: Replica: fea1cb04b3954e36952f2144017d4d8c, State: Running, Role: LEADER
I20260929 16:26:30.877099 7845 consensus_queue.cc:237] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: false } }
I20260929 16:26:30.878285 7848 sys_catalog.cc:455] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fea1cb04b3954e36952f2144017d4d8c. Latest consensus state: current_term: 3 leader_uuid: "fea1cb04b3954e36952f2144017d4d8c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: false } } }
I20260929 16:26:30.878405 7848 sys_catalog.cc:458] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:30.878675 7850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:26:30.880812 7850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:26:30.881254 7850 catalog_manager.cc:1259] Loaded cluster ID: d3fbbee867e94a12b43864bbe215c0dc
I20260929 16:26:30.881311 7850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:26:30.881873 7850 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:26:30.882246 7850 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c: Loaded TSK: 0
I20260929 16:26:30.882908 7850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:26:30.930938 7304 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:26:30.934451 7304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:26:30.937671 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:30.937398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.0.1:44106)
0929 16:26:30.937535 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:30.937597 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 16:26:30.937780 7134 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [term 3 FOLLOWER]: Refusing update from remote peer fea1cb04b3954e36952f2144017d4d8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260929 16:26:30.938356 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:30.938129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.0.1:44116)
0929 16:26:30.938232 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:30.938284 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 16:26:30.938412 7845 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 16:26:30.938627 7292 consensus_peers.cc:487] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c -> Peer 6d732cddd2a8490ea5a12bdcc1798761 (127.0.98.62:44555): Couldn't send request to peer 6d732cddd2a8490ea5a12bdcc1798761. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 16:26:30.940162 7852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fea1cb04b3954e36952f2144017d4d8c. Latest consensus state: current_term: 3 leader_uuid: "fea1cb04b3954e36952f2144017d4d8c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: false } } }
I20260929 16:26:30.940299 7852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:26:30.941519 7845 sys_catalog.cc:455] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "fea1cb04b3954e36952f2144017d4d8c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: false } } }
I20260929 16:26:30.941642 7845 sys_catalog.cc:458] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:30.941915 7852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "fea1cb04b3954e36952f2144017d4d8c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6d732cddd2a8490ea5a12bdcc1798761" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44555 } } peers { permanent_uuid: "7bc65de05f0c4d39a89442538492b338" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36225 } attrs { promote: false } } peers { permanent_uuid: "fea1cb04b3954e36952f2144017d4d8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33633 } attrs { promote: false } } }
I20260929 16:26:30.942116 7852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bc65de05f0c4d39a89442538492b338 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:26:30.952332 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:30.948192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43549 (local address 127.0.0.1:33338)
0929 16:26:30.948349 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:30.948364 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:30.948456 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:30.948968 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:30.948974 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:30.948998 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:30.949121 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:30.949129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:30.950262 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:30.950267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:30.951061 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:30.951069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:30.951957 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:30.951961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:30.952076 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:30.952108 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:26:30.952158 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 16:26:30.952223 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":2}
I20260929 16:26:30.952879 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:30.948321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.0.98.2:43549)
0929 16:26:30.948692 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:30.948709 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:30.948725 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:26:30.948782 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:30.948787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:30.948843 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:30.949283 (+ 440us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:30.949405 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:30.950170 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:30.951185 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:30.951795 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:30.952559 (+ 764us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:30.952622 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:30.952659 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 16:26:30.952738 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":95,"threads_started":1}
I20260929 16:26:30.953488 7557 tablet_service.cc:1467] Processing CreateTablet for tablet 0b75ab0833a34c0598bed46e7a3968aa (DEFAULT_TABLE table=test-table [id=11be0a0214fd4dabbc3c976daa95907d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:30.953558 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:30.948584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46539 (local address 127.0.0.1:36392)
0929 16:26:30.949251 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:30.949266 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:30.949359 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:30.950114 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:30.950119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:30.950137 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:30.950212 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:30.950218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:30.951256 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:30.951259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:30.952415 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:30.952424 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:30.953126 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:30.953130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:30.953260 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:30.953283 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:26:30.953359 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 16:26:30.953426 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":16,"thread_start_us":90,"threads_started":1}
I20260929 16:26:30.953619 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:30.948935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36392 (local address 127.0.98.3:46539)
0929 16:26:30.949903 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:30.949908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:30.949921 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:26:30.949969 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:30.949973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:30.950014 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:30.950102 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:30.950370 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:30.951108 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:30.952547 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:30.953016 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:30.953131 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:30.953234 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:30.953388 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 16:26:30.953465 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":73,"threads_started":1}
I20260929 16:26:30.953827 7557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b75ab0833a34c0598bed46e7a3968aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:30.954108 7693 tablet_service.cc:1467] Processing CreateTablet for tablet 0b75ab0833a34c0598bed46e7a3968aa (DEFAULT_TABLE table=test-table [id=11be0a0214fd4dabbc3c976daa95907d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:30.954378 7693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b75ab0833a34c0598bed46e7a3968aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:30.954548 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:30.948897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41403 (local address 127.0.0.1:41396)
0929 16:26:30.949577 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:30.949591 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:30.949691 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:30.949912 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:30.949915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:30.949930 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:30.950019 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:30.950024 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:30.951064 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:30.951067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:30.952320 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:30.952330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:30.954080 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:30.954083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:30.954249 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:30.954273 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:26:30.954355 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 16:26:30.954428 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":26,"thread_start_us":114,"threads_started":1}
I20260929 16:26:30.954978 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:30.949315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41396 (local address 127.0.98.1:41403)
0929 16:26:30.949542 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:30.949545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:30.949666 (+ 121us) server_negotiation.cc:407] Connection header received
0929 16:26:30.949775 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:30.949777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:30.949813 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:30.949904 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:30.950162 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:30.950941 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:30.952707 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:30.953956 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:30.954067 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:30.954129 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:30.954390 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 16:26:30.954519 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":96,"threads_started":1}
I20260929 16:26:30.955781 7420 tablet_service.cc:1467] Processing CreateTablet for tablet 0b75ab0833a34c0598bed46e7a3968aa (DEFAULT_TABLE table=test-table [id=11be0a0214fd4dabbc3c976daa95907d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:30.956048 7420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b75ab0833a34c0598bed46e7a3968aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:30.956653 7863 tablet_bootstrap.cc:492] T 0b75ab0833a34c0598bed46e7a3968aa P bcbdbf5a43df466cb16a8911c91e24a6: Bootstrap starting.
I20260929 16:26:30.957118 7864 tablet_bootstrap.cc:492] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7: Bootstrap starting.
I20260929 16:26:30.957755 7863 tablet_bootstrap.cc:654] T 0b75ab0833a34c0598bed46e7a3968aa P bcbdbf5a43df466cb16a8911c91e24a6: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:30.958078 7864 tablet_bootstrap.cc:654] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:30.959370 7864 tablet_bootstrap.cc:492] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7: No bootstrap required, opened a new log
I20260929 16:26:30.959462 7864 ts_tablet_manager.cc:1397] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:26:30.959699 7863 tablet_bootstrap.cc:492] T 0b75ab0833a34c0598bed46e7a3968aa P bcbdbf5a43df466cb16a8911c91e24a6: No bootstrap required, opened a new log
I20260929 16:26:30.959797 7863 ts_tablet_manager.cc:1397] T 0b75ab0833a34c0598bed46e7a3968aa P bcbdbf5a43df466cb16a8911c91e24a6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 16:26:30.959909 7864 raft_consensus.cc:348] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43549 } } peers { permanent_uuid: "4bcbbb5789854afbbea071a979eccfb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46539 } } peers { permanent_uuid: "13adcbd1e74248ab871d13068c0f23c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41403 } }
I20260929 16:26:30.960023 7864 raft_consensus.cc:374] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:30.960047 7864 raft_consensus.cc:729] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcbbb5789854afbbea071a979eccfb7, State: Initialized, Role: FOLLOWER
I20260929 16:26:30.960203 7864 consensus_queue.cc:260] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43549 } } peers { permanent_uuid: "4bcbbb5789854afbbea071a979eccfb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46539 } } peers { permanent_uuid: "13adcbd1e74248ab871d13068c0f23c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41403 } }
I20260929 16:26:30.960213 7863 raft_consensus.cc:348] T 0b75ab0833a34c0598bed46e7a3968aa P bcbdbf5a43df466cb16a8911c91e24a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43549 } } peers { permanent_uuid: "4bcbbb5789854afbbea071a979eccfb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46539 } } peers { permanent_uuid: "13adcbd1e74248ab871d13068c0f23c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41403 } }
I20260929 16:26:30.960352 7863 raft_consensus.cc:374] T 0b75ab0833a34c0598bed46e7a3968aa P bcbdbf5a43df466cb16a8911c91e24a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:30.960390 7863 raft_consensus.cc:729] T 0b75ab0833a34c0598bed46e7a3968aa P bcbdbf5a43df466cb16a8911c91e24a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcbdbf5a43df466cb16a8911c91e24a6, State: Initialized, Role: FOLLOWER
I20260929 16:26:30.960517 7863 consensus_queue.cc:260] T 0b75ab0833a34c0598bed46e7a3968aa P bcbdbf5a43df466cb16a8911c91e24a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43549 } } peers { permanent_uuid: "4bcbbb5789854afbbea071a979eccfb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46539 } } peers { permanent_uuid: "13adcbd1e74248ab871d13068c0f23c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41403 } }
I20260929 16:26:30.960569 7864 ts_tablet_manager.cc:1428] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 16:26:30.960867 7863 ts_tablet_manager.cc:1428] T 0b75ab0833a34c0598bed46e7a3968aa P bcbdbf5a43df466cb16a8911c91e24a6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 16:26:30.961146 7866 tablet_bootstrap.cc:492] T 0b75ab0833a34c0598bed46e7a3968aa P 13adcbd1e74248ab871d13068c0f23c9: Bootstrap starting.
I20260929 16:26:30.962119 7866 tablet_bootstrap.cc:654] T 0b75ab0833a34c0598bed46e7a3968aa P 13adcbd1e74248ab871d13068c0f23c9: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:30.962519 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:30.961697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.3:55065)
0929 16:26:30.962412 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:30.962462 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":638}
I20260929 16:26:30.962810 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:30.961932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.2:39557)
0929 16:26:30.962708 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:30.962759 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":698}
I20260929 16:26:30.963516 7623 heartbeater.cc:503] Master 127.0.98.60:33633 was elected leader, sending a full tablet report...
I20260929 16:26:30.963765 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:30.963557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.3:41783)
0929 16:26:30.963647 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:30.963705 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 16:26:30.964160 7759 heartbeater.cc:503] Master 127.0.98.60:33633 was elected leader, sending a full tablet report...
I20260929 16:26:30.965049 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:30.964822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.2:36793)
0929 16:26:30.964962 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:30.964999 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 16:26:30.965883 7866 tablet_bootstrap.cc:492] T 0b75ab0833a34c0598bed46e7a3968aa P 13adcbd1e74248ab871d13068c0f23c9: No bootstrap required, opened a new log
I20260929 16:26:30.965981 7866 ts_tablet_manager.cc:1397] T 0b75ab0833a34c0598bed46e7a3968aa P 13adcbd1e74248ab871d13068c0f23c9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 16:26:30.966384 7866 raft_consensus.cc:348] T 0b75ab0833a34c0598bed46e7a3968aa P 13adcbd1e74248ab871d13068c0f23c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43549 } } peers { permanent_uuid: "4bcbbb5789854afbbea071a979eccfb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46539 } } peers { permanent_uuid: "13adcbd1e74248ab871d13068c0f23c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41403 } }
I20260929 16:26:30.966476 7866 raft_consensus.cc:374] T 0b75ab0833a34c0598bed46e7a3968aa P 13adcbd1e74248ab871d13068c0f23c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:30.966526 7866 raft_consensus.cc:729] T 0b75ab0833a34c0598bed46e7a3968aa P 13adcbd1e74248ab871d13068c0f23c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13adcbd1e74248ab871d13068c0f23c9, State: Initialized, Role: FOLLOWER
I20260929 16:26:30.966679 7866 consensus_queue.cc:260] T 0b75ab0833a34c0598bed46e7a3968aa P 13adcbd1e74248ab871d13068c0f23c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43549 } } peers { permanent_uuid: "4bcbbb5789854afbbea071a979eccfb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46539 } } peers { permanent_uuid: "13adcbd1e74248ab871d13068c0f23c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41403 } }
I20260929 16:26:30.967427 7866 ts_tablet_manager.cc:1428] T 0b75ab0833a34c0598bed46e7a3968aa P 13adcbd1e74248ab871d13068c0f23c9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 16:26:30.967617 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:30.967301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.1:38481)
0929 16:26:30.967504 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:30.967556 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260929 16:26:30.968322 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:30.968075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.1:55829)
0929 16:26:30.968194 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:30.968242 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 16:26:30.968515 7486 heartbeater.cc:503] Master 127.0.98.60:33633 was elected leader, sending a full tablet report...
W20260929 16:26:31.071167 7762 tablet.cc:2369] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:26:31.076189 7868 raft_consensus.cc:482] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:26:31.076344 7868 raft_consensus.cc:504] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43549 } } peers { permanent_uuid: "4bcbbb5789854afbbea071a979eccfb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46539 } } peers { permanent_uuid: "13adcbd1e74248ab871d13068c0f23c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41403 } }
I20260929 16:26:31.077549 7868 leader_election.cc:288] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcbdbf5a43df466cb16a8911c91e24a6 (127.0.98.2:43549), 13adcbd1e74248ab871d13068c0f23c9 (127.0.98.1:41403)
I20260929 16:26:31.081676 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:31.078000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43295 (local address 127.0.98.2:43549)
0929 16:26:31.078129 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:31.078133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:31.078413 (+ 280us) server_negotiation.cc:407] Connection header received
0929 16:26:31.078553 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:31.078556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:31.078608 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:31.078692 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:31.078937 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:31.079437 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:31.080603 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:31.081018 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:31.081147 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:31.081242 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:31.081470 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 16:26:31.081550 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:26:31.081693 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:31.077989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59673 (local address 127.0.98.1:41403)
0929 16:26:31.078116 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:31.078120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:31.078138 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:26:31.078352 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:31.078356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:31.078431 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:31.078539 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:31.078870 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:31.079445 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:31.080610 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:31.081043 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:31.081164 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:31.081265 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:31.081489 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 16:26:31.081566 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:26:31.081686 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:31.077841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41403 (local address 127.0.98.3:59673)
0929 16:26:31.078051 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:31.078069 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:31.078186 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:31.078565 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:31.078568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:31.078586 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:31.078723 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:31.078730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:31.079561 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:31.079567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:31.080479 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:31.080488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:31.081167 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:31.081171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:31.081363 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:31.081381 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 16:26:31.081472 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 16:26:31.081546 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":64}
I20260929 16:26:31.081691 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:31.077865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43549 (local address 127.0.98.3:43295)
0929 16:26:31.078310 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:31.078345 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:31.078445 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:31.078703 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:31.078706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:31.078722 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:31.078825 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:31.078830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:31.079550 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:31.079553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:31.080481 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:31.080490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:31.081163 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:31.081167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:31.081337 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:31.081364 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:26:31.081454 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 16:26:31.081526 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":78,"thread_start_us":122,"threads_started":1}
I20260929 16:26:31.082356 7440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b75ab0833a34c0598bed46e7a3968aa" candidate_uuid: "4bcbbb5789854afbbea071a979eccfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13adcbd1e74248ab871d13068c0f23c9" is_pre_election: true
I20260929 16:26:31.082515 7440 raft_consensus.cc:2454] T 0b75ab0833a34c0598bed46e7a3968aa P 13adcbd1e74248ab871d13068c0f23c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bcbbb5789854afbbea071a979eccfb7 in term 0.
I20260929 16:26:31.082471 7577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b75ab0833a34c0598bed46e7a3968aa" candidate_uuid: "4bcbbb5789854afbbea071a979eccfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" is_pre_election: true
I20260929 16:26:31.082734 7577 raft_consensus.cc:2454] T 0b75ab0833a34c0598bed46e7a3968aa P bcbdbf5a43df466cb16a8911c91e24a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bcbbb5789854afbbea071a979eccfb7 in term 0.
I20260929 16:26:31.082926 7648 leader_election.cc:302] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13adcbd1e74248ab871d13068c0f23c9, 4bcbbb5789854afbbea071a979eccfb7; no voters:
I20260929 16:26:31.083217 7868 raft_consensus.cc:2781] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:26:31.083293 7868 raft_consensus.cc:482] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:26:31.083348 7868 raft_consensus.cc:3037] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:31.084403 7868 raft_consensus.cc:504] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43549 } } peers { permanent_uuid: "4bcbbb5789854afbbea071a979eccfb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46539 } } peers { permanent_uuid: "13adcbd1e74248ab871d13068c0f23c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41403 } }
I20260929 16:26:31.084789 7868 leader_election.cc:288] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7 [CANDIDATE]: Term 1 election: Requested vote from peers bcbdbf5a43df466cb16a8911c91e24a6 (127.0.98.2:43549), 13adcbd1e74248ab871d13068c0f23c9 (127.0.98.1:41403)
I20260929 16:26:31.085114 7577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b75ab0833a34c0598bed46e7a3968aa" candidate_uuid: "4bcbbb5789854afbbea071a979eccfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcbdbf5a43df466cb16a8911c91e24a6"
I20260929 16:26:31.085278 7577 raft_consensus.cc:3037] T 0b75ab0833a34c0598bed46e7a3968aa P bcbdbf5a43df466cb16a8911c91e24a6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:31.085270 7440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b75ab0833a34c0598bed46e7a3968aa" candidate_uuid: "4bcbbb5789854afbbea071a979eccfb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13adcbd1e74248ab871d13068c0f23c9"
I20260929 16:26:31.085409 7440 raft_consensus.cc:3037] T 0b75ab0833a34c0598bed46e7a3968aa P 13adcbd1e74248ab871d13068c0f23c9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:31.086259 7577 raft_consensus.cc:2454] T 0b75ab0833a34c0598bed46e7a3968aa P bcbdbf5a43df466cb16a8911c91e24a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bcbbb5789854afbbea071a979eccfb7 in term 1.
I20260929 16:26:31.086391 7440 raft_consensus.cc:2454] T 0b75ab0833a34c0598bed46e7a3968aa P 13adcbd1e74248ab871d13068c0f23c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bcbbb5789854afbbea071a979eccfb7 in term 1.
I20260929 16:26:31.086594 7646 leader_election.cc:302] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcbbb5789854afbbea071a979eccfb7, bcbdbf5a43df466cb16a8911c91e24a6; no voters:
I20260929 16:26:31.086750 7868 raft_consensus.cc:2781] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:31.086968 7868 raft_consensus.cc:686] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7 [term 1 LEADER]: Becoming Leader. State: Replica: 4bcbbb5789854afbbea071a979eccfb7, State: Running, Role: LEADER
I20260929 16:26:31.087311 7868 consensus_queue.cc:237] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43549 } } peers { permanent_uuid: "4bcbbb5789854afbbea071a979eccfb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46539 } } peers { permanent_uuid: "13adcbd1e74248ab871d13068c0f23c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41403 } }
I20260929 16:26:31.087735 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:31.087492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.3:39629)
0929 16:26:31.087599 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:31.087661 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 16:26:31.089761 7306 catalog_manager.cc:5495] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bcbbb5789854afbbea071a979eccfb7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4bcbbb5789854afbbea071a979eccfb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bcbbb5789854afbbea071a979eccfb7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13adcbd1e74248ab871d13068c0f23c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41403 } health_report { overall_health: UNKNOWN } } }
I20260929 16:26:31.107327 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:31.103948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36394 (local address 127.0.98.3:46539)
0929 16:26:31.104068 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:31.104072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:31.104086 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:26:31.104213 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:31.104215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:31.104298 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:31.104403 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:31.104655 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:31.105157 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:31.105995 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:31.106212 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:31.106289 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:31.106522 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:31.106971 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:31.107060 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:31.107171 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 16:26:31.107220 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 16:26:31.110128 7440 raft_consensus.cc:1261] T 0b75ab0833a34c0598bed46e7a3968aa P 13adcbd1e74248ab871d13068c0f23c9 [term 1 FOLLOWER]: Refusing update from remote peer 4bcbbb5789854afbbea071a979eccfb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:31.110626 7868 consensus_queue.cc:1035] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13adcbd1e74248ab871d13068c0f23c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:31.110858 7577 raft_consensus.cc:1261] T 0b75ab0833a34c0598bed46e7a3968aa P bcbdbf5a43df466cb16a8911c91e24a6 [term 1 FOLLOWER]: Refusing update from remote peer 4bcbbb5789854afbbea071a979eccfb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:31.111444 7868 consensus_queue.cc:1035] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcbdbf5a43df466cb16a8911c91e24a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:31.112474 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:31.112147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.1:51981)
0929 16:26:31.112306 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:31.112395 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 16:26:31.113310 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:31.112978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.2:58531)
0929 16:26:31.113136 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:31.113195 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 16:26:31.115335 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:31.114806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.3:59403)
0929 16:26:31.115201 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:31.115259 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325}
W20260929 16:26:31.115656 7626 tablet.cc:2369] T 0b75ab0833a34c0598bed46e7a3968aa P bcbdbf5a43df466cb16a8911c91e24a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:26:31.116330 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:31.116018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.1:43797)
0929 16:26:31.116201 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:31.116260 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260929 16:26:31.118257 7877 mvcc.cc:205] Tried to move back new op lower bound from 7334703886781833216 to 7334703886696493056. Current Snapshot: MvccSnapshot[applied={T|T < 7334703886781833216}]
I20260929 16:26:31.119524 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:31.119144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.3:39279)
0929 16:26:31.119360 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:31.119430 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260929 16:26:31.120447 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:31.120057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44555 (local address 127.0.98.2:52159)
0929 16:26:31.120302 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:31.120364 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260929 16:26:31.121445 7879 mvcc.cc:205] Tried to move back new op lower bound from 7334703886781833216 to 7334703886696493056. Current Snapshot: MvccSnapshot[applied={T|T < 7334703886781833216}]
I20260929 16:26:31.124992 7305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:31.125134 7305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:31.129086 7305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fea1cb04b3954e36952f2144017d4d8c: Sending DeleteTablet for 3 replicas of tablet 0b75ab0833a34c0598bed46e7a3968aa
I20260929 16:26:31.129769 7557 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b75ab0833a34c0598bed46e7a3968aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:31 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33338
I20260929 16:26:31.129936 7693 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b75ab0833a34c0598bed46e7a3968aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:31 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36392
I20260929 16:26:31.130230 7885 tablet_replica.cc:330] T 0b75ab0833a34c0598bed46e7a3968aa P bcbdbf5a43df466cb16a8911c91e24a6: stopping tablet replica
I20260929 16:26:31.130466 7420 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b75ab0833a34c0598bed46e7a3968aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:31 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41396
I20260929 16:26:31.130813 7887 tablet_replica.cc:330] T 0b75ab0833a34c0598bed46e7a3968aa P 13adcbd1e74248ab871d13068c0f23c9: stopping tablet replica
I20260929 16:26:31.130864 7885 raft_consensus.cc:2229] T 0b75ab0833a34c0598bed46e7a3968aa P bcbdbf5a43df466cb16a8911c91e24a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:31.130944 7887 raft_consensus.cc:2229] T 0b75ab0833a34c0598bed46e7a3968aa P 13adcbd1e74248ab871d13068c0f23c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:31.131079 7887 raft_consensus.cc:2258] T 0b75ab0833a34c0598bed46e7a3968aa P 13adcbd1e74248ab871d13068c0f23c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:31.131119 7885 raft_consensus.cc:2258] T 0b75ab0833a34c0598bed46e7a3968aa P bcbdbf5a43df466cb16a8911c91e24a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:31.131258 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 7268
I20260929 16:26:31.133155 7885 ts_tablet_manager.cc:1905] T 0b75ab0833a34c0598bed46e7a3968aa P bcbdbf5a43df466cb16a8911c91e24a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:31.136669 7885 ts_tablet_manager.cc:1918] T 0b75ab0833a34c0598bed46e7a3968aa P bcbdbf5a43df466cb16a8911c91e24a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:26:31.136785 7885 log.cc:1196] T 0b75ab0833a34c0598bed46e7a3968aa P bcbdbf5a43df466cb16a8911c91e24a6: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/0b75ab0833a34c0598bed46e7a3968aa
I20260929 16:26:31.137085 7885 ts_tablet_manager.cc:1939] T 0b75ab0833a34c0598bed46e7a3968aa P bcbdbf5a43df466cb16a8911c91e24a6: Deleting consensus metadata
I20260929 16:26:31.137113 7886 tablet_replica.cc:330] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7: stopping tablet replica
I20260929 16:26:31.137385 7886 raft_consensus.cc:2229] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:26:31.137782 7886 raft_consensus.cc:2258] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:31.138017 7291 catalog_manager.cc:4841] TS bcbdbf5a43df466cb16a8911c91e24a6 (127.0.98.2:43549): tablet 0b75ab0833a34c0598bed46e7a3968aa (table test-table [id=11be0a0214fd4dabbc3c976daa95907d]) successfully deleted
I20260929 16:26:31.138705 7886 ts_tablet_manager.cc:1905] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 16:26:31.139760 7861 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:31.139042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41403 (local address 127.0.0.1:41400)
0929 16:26:31.139151 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:31.139180 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:31.139271 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:31.139617 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41403: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 16:26:31.139885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 7494
I20260929 16:26:31.141517 7886 ts_tablet_manager.cc:1918] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:26:31.141600 7886 log.cc:1196] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/0b75ab0833a34c0598bed46e7a3968aa
I20260929 16:26:31.141887 7886 ts_tablet_manager.cc:1939] T 0b75ab0833a34c0598bed46e7a3968aa P 4bcbbb5789854afbbea071a979eccfb7: Deleting consensus metadata
W20260929 16:26:31.142215 7293 catalog_manager.cc:4584] TS 13adcbd1e74248ab871d13068c0f23c9 (127.0.98.1:41403): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0b75ab0833a34c0598bed46e7a3968aa: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41403: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 16:26:31.143069 7294 catalog_manager.cc:4841] TS 4bcbbb5789854afbbea071a979eccfb7 (127.0.98.3:46539): tablet 0b75ab0833a34c0598bed46e7a3968aa (table test-table [id=11be0a0214fd4dabbc3c976daa95907d]) successfully deleted
I20260929 16:26:31.147622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 7630
I20260929 16:26:31.153314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 6980
I20260929 16:26:31.159122 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 7089
2026-09-29T16:26:31Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7087 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:26:31 dist-test-slave-vq9v krb5kdc[7890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:26:31 dist-test-slave-vq9v krb5kdc[7890](info): set up 2 sockets
Sep 29 16:26:31 dist-test-slave-vq9v krb5kdc[7890](info): commencing operation
krb5kdc: starting...
W20260929 16:26:32.202075 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:26:32 dist-test-slave-vq9v krb5kdc[7890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:26:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:26:32Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260929 16:26:32.332562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38557
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:32.445393 7903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:32.445689 7903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:32.445732 7903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:32.449828 7903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:26:32.449896 7903 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 16:26:32.449914 7903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:32.449934 7903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:32.449950 7903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:26:32.450004 7903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:26:32.455499 7903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38557
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:32.456669 7903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:32.457742 7903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:32.466724 7911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:32.466724 7908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:32.467061 7903 server_base.cc:1034] running on GCE node
W20260929 16:26:32.466730 7909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:32.467693 7903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:32.468973 7903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:32.470156 7903 hybrid_clock.cc:648] HybridClock initialized: now 1790699192470129 us; error 44 us; skew 500 ppm
Sep 29 16:26:32 dist-test-slave-vq9v krb5kdc[7890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699192, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:32.473044 7903 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260929 16:26:32.474007 7903 webserver.cc:460] Webserver started at http://127.0.98.62:46733/ using document root <none> and password file <none>
I20260929 16:26:32.474581 7903 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:32.474677 7903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:32.474901 7903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:32.476972 7903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "29fc8946cc8b4a299a1be3ef8cc4815b"
format_stamp: "Formatted at 2026-09-29 16:26:32 on dist-test-slave-vq9v"
server_key: "7e8f8a6c36165c567ece7d934dc62696"
server_key_iv: "72725b9ebc050b46c349ab75cae68422"
server_key_version: "encryptionkey@0"
I20260929 16:26:32.477591 7903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "29fc8946cc8b4a299a1be3ef8cc4815b"
format_stamp: "Formatted at 2026-09-29 16:26:32 on dist-test-slave-vq9v"
server_key: "7e8f8a6c36165c567ece7d934dc62696"
server_key_iv: "72725b9ebc050b46c349ab75cae68422"
server_key_version: "encryptionkey@0"
I20260929 16:26:32.481597 7903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:26:32.484213 7918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:32.485574 7903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:32.485716 7903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "29fc8946cc8b4a299a1be3ef8cc4815b"
format_stamp: "Formatted at 2026-09-29 16:26:32 on dist-test-slave-vq9v"
server_key: "7e8f8a6c36165c567ece7d934dc62696"
server_key_iv: "72725b9ebc050b46c349ab75cae68422"
server_key_version: "encryptionkey@0"
I20260929 16:26:32.485852 7903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:32.499758 7903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:32.503048 7903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:32.503274 7903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:32.511088 7903 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38557
I20260929 16:26:32.511116 7980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38557 every 8 connection(s)
I20260929 16:26:32.512068 7903 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:26:32.514812 7981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:32.519284 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 7903
I20260929 16:26:32.519410 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:26:32.519732 392 external_mini_cluster.cc:1428] Setting key 54a5a0461c3c767c54e457b967ec0cbc
I20260929 16:26:32.520782 7981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b: Bootstrap starting.
I20260929 16:26:32.523582 7981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:32.524431 7981 log.cc:824] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:32.526613 7981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b: No bootstrap required, opened a new log
Sep 29 16:26:32 dist-test-slave-vq9v krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 16:26:32.531006 7981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fc8946cc8b4a299a1be3ef8cc4815b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } }
I20260929 16:26:32.531337 7981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:32.531416 7981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29fc8946cc8b4a299a1be3ef8cc4815b, State: Initialized, Role: FOLLOWER
I20260929 16:26:32.531961 7981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fc8946cc8b4a299a1be3ef8cc4815b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } }
I20260929 16:26:32.532150 7981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:26:32.532219 7981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:26:32.532316 7981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:32.533506 7981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fc8946cc8b4a299a1be3ef8cc4815b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } }
I20260929 16:26:32.533874 7981 leader_election.cc:302] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29fc8946cc8b4a299a1be3ef8cc4815b; no voters:
I20260929 16:26:32.534214 7981 leader_election.cc:288] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:26:32.534344 7986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:32.534574 7986 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b [term 1 LEADER]: Becoming Leader. State: Replica: 29fc8946cc8b4a299a1be3ef8cc4815b, State: Running, Role: LEADER
I20260929 16:26:32.534926 7986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fc8946cc8b4a299a1be3ef8cc4815b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } }
I20260929 16:26:32.535558 7981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:26:32.536098 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:32.521340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53030 (local address 127.0.98.62:38557)
0929 16:26:32.521778 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:32.521797 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:32.521840 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:26:32.522729 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:32.522754 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:32.523204 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:32.523597 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:32.524511 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:32.525895 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:32.526872 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:32.527552 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:32.527718 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:32.530556 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:32.530563 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:32.530580 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:32.530612 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:32.532650 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:32.533073 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:32.533079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:32.533085 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:32.533259 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:32.533519 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:32.533524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:32.533525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:32.534069 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:32.534288 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:32.534629 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 16:26:32.535104 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":138,"threads_started":1}
I20260929 16:26:32.536182 7986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29fc8946cc8b4a299a1be3ef8cc4815b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fc8946cc8b4a299a1be3ef8cc4815b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } } }
I20260929 16:26:32.536302 7986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:32.536489 7986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29fc8946cc8b4a299a1be3ef8cc4815b. Latest consensus state: current_term: 1 leader_uuid: "29fc8946cc8b4a299a1be3ef8cc4815b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fc8946cc8b4a299a1be3ef8cc4815b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38557 } } }
I20260929 16:26:32.536563 7986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:32.536854 7993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:26:32.540001 7993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:26:32.546326 7993 catalog_manager.cc:1347] Generated new cluster ID: 649d4733acb64553898c38c063602dca
I20260929 16:26:32.546418 7993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:26:32.567893 7993 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:26:32.568907 7993 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:26:32.581921 7993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b: Generated new TSK 0
I20260929 16:26:32.582753 7993 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260929 16:26:32.621639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38557
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:32.757588 8007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:32.757961 8007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:32.758047 8007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:32.762352 8007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:32.762426 8007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:32.762468 8007 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:26:32.762542 8007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:26:32.768072 8007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:32.769222 8007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:32.770267 8007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:26:32.780121 8007 server_base.cc:1034] running on GCE node
W20260929 16:26:32.780061 8013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:32.780066 8012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:32.780066 8015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:32.780748 8007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:32.781508 8007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:32.782665 8007 hybrid_clock.cc:648] HybridClock initialized: now 1790699192782648 us; error 31 us; skew 500 ppm
Sep 29 16:26:32 dist-test-slave-vq9v krb5kdc[7890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699192, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:32.785575 8007 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 16:26:32.786423 8007 webserver.cc:460] Webserver started at http://127.0.98.1:44687/ using document root <none> and password file <none>
I20260929 16:26:32.786962 8007 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:32.787024 8007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:32.787192 8007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:32.789275 8007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57"
format_stamp: "Formatted at 2026-09-29 16:26:32 on dist-test-slave-vq9v"
server_key: "8aa6f2105b2bcf56cb1e664ecc29179b"
server_key_iv: "710d97dc39a1be52e4d37e737915fa3a"
server_key_version: "encryptionkey@0"
I20260929 16:26:32.789871 8007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57"
format_stamp: "Formatted at 2026-09-29 16:26:32 on dist-test-slave-vq9v"
server_key: "8aa6f2105b2bcf56cb1e664ecc29179b"
server_key_iv: "710d97dc39a1be52e4d37e737915fa3a"
server_key_version: "encryptionkey@0"
I20260929 16:26:32.793893 8007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:26:32.796590 8022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:32.798086 8007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:32.798213 8007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57"
format_stamp: "Formatted at 2026-09-29 16:26:32 on dist-test-slave-vq9v"
server_key: "8aa6f2105b2bcf56cb1e664ecc29179b"
server_key_iv: "710d97dc39a1be52e4d37e737915fa3a"
server_key_version: "encryptionkey@0"
I20260929 16:26:32.798324 8007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:32.811293 8007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:32.814877 8007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:32.815109 8007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:32.816895 8007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:32.816951 8007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:32.816993 8007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:32.817044 8007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:26:32 dist-test-slave-vq9v krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699192, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 16:26:32.827980 8007 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46171
I20260929 16:26:32.827998 8137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46171 every 8 connection(s)
I20260929 16:26:32.828958 8007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:26:32.829241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 8007
I20260929 16:26:32.829401 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:26:32.829698 392 external_mini_cluster.cc:1428] Setting key a08cd83a7101e57ce1344c64e6033db1
I20260929 16:26:32.834537 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:32.817785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48823 (local address 127.0.98.62:38557)
0929 16:26:32.817970 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:32.817976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:32.818863 (+ 887us) server_negotiation.cc:407] Connection header received
0929 16:26:32.820040 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:32.820046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:32.820119 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:32.820239 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:32.821977 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:32.822881 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:32.824780 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:32.825033 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:32.825149 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:32.828679 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:32.828684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:32.828687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:32.828721 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:32.831962 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:32.832676 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:32.832679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:32.832680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:32.832767 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:32.833237 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:32.833242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:32.833246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:32.833593 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:32.833718 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:32.834237 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 16:26:32.834404 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:26:32.835331 8031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:32.818211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:48823)
0929 16:26:32.818673 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:32.818725 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:32.819776 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:32.820454 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:32.820466 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:32.820954 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:32.821742 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:32.821766 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:32.823054 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:32.823057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:32.824638 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:32.824651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:32.825192 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:32.825196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:32.825430 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:32.826145 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:32.826164 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:32.828458 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:32.832091 (+ 3633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:32.832100 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:32.832121 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:32.832551 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:32.832861 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:32.832867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:32.832872 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:32.833096 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:32.833707 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:32.833713 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:32.833985 (+ 272us) client_negotiation.cc:769] Sending connection context
0929 16:26:32.834151 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 16:26:32.834426 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":132,"threads_started":1}
I20260929 16:26:32.836882 8138 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38557
I20260929 16:26:32.837065 8138 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:32.837760 8138 heartbeater.cc:511] Master 127.0.98.62:38557 requested a full tablet report, sending...
I20260929 16:26:32.839583 7934 ts_manager.cc:194] Registered new tserver with Master: a3c12d4976bc4aa3a67d4ff1a5ab7a57 (127.0.98.1:46171)
I20260929 16:26:32.841405 7934 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48823
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260929 16:26:32.847959 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:32.839167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.1:38789)
0929 16:26:32.839542 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:32.839555 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:32.839672 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:32.840028 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:32.840030 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:32.840228 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:32.840344 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:32.840349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:32.841298 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:32.841302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:32.842063 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:32.842072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:32.842735 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:32.842740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:32.842853 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:32.843520 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:32.843536 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:32.844030 (+ 494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:32.846011 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:32.846013 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:32.846015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:32.846301 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:32.846629 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:32.846635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:32.846640 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:32.846763 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:32.847251 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:32.847257 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:32.847378 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 16:26:32.847506 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 16:26:32.847691 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":109,"threads_started":1}
I20260929 16:26:32.847999 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:32.839208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38789 (local address 127.0.98.62:38557)
0929 16:26:32.839405 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:32.839410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:32.839625 (+ 215us) server_negotiation.cc:407] Connection header received
0929 16:26:32.839792 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:32.839795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:32.839855 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:32.839900 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:32.840464 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:32.841133 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:32.842205 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:32.842590 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:32.842734 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:32.844182 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:32.844186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:32.844187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:32.844210 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:32.845876 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:32.846441 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:32.846444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:32.846446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:32.846527 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:32.846884 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:32.846888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:32.846889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:32.847096 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:32.847259 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:32.847613 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 16:26:32.847782 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260929 16:26:32.865509 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38557
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:32.978804 8147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:32.979074 8147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:32.979122 8147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:32.983309 8147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:32.983377 8147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:32.983418 8147 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:26:32.983453 8147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:26:32.988867 8147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:32.990025 8147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:32.991015 8147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:33.001363 8155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:33.001363 8153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:33.001431 8147 server_base.cc:1034] running on GCE node
W20260929 16:26:33.001441 8152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:33.002208 8147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:33.003113 8147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:33.004326 8147 hybrid_clock.cc:648] HybridClock initialized: now 1790699193004284 us; error 56 us; skew 500 ppm
Sep 29 16:26:33 dist-test-slave-vq9v krb5kdc[7890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:33.007638 8147 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 16:26:33.008618 8147 webserver.cc:460] Webserver started at http://127.0.98.2:44875/ using document root <none> and password file <none>
I20260929 16:26:33.009291 8147 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:33.009385 8147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:33.009620 8147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:33.011739 8147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "2ba27a37ecd048cf95328fc90b9b6565"
format_stamp: "Formatted at 2026-09-29 16:26:33 on dist-test-slave-vq9v"
server_key: "0fb8c7889d064386a915eb36d96ed8f8"
server_key_iv: "776d0bdcfd88acc6a4630069debb0798"
server_key_version: "encryptionkey@0"
I20260929 16:26:33.012326 8147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2ba27a37ecd048cf95328fc90b9b6565"
format_stamp: "Formatted at 2026-09-29 16:26:33 on dist-test-slave-vq9v"
server_key: "0fb8c7889d064386a915eb36d96ed8f8"
server_key_iv: "776d0bdcfd88acc6a4630069debb0798"
server_key_version: "encryptionkey@0"
I20260929 16:26:33.016211 8147 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 16:26:33.019006 8162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:33.020386 8147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:26:33.020530 8147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "2ba27a37ecd048cf95328fc90b9b6565"
format_stamp: "Formatted at 2026-09-29 16:26:33 on dist-test-slave-vq9v"
server_key: "0fb8c7889d064386a915eb36d96ed8f8"
server_key_iv: "776d0bdcfd88acc6a4630069debb0798"
server_key_version: "encryptionkey@0"
I20260929 16:26:33.020668 8147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:33.032394 8147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:33.035763 8147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:33.035993 8147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:33.037925 8147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:33.038003 8147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:33.038080 8147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:33.038123 8147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:26:33 dist-test-slave-vq9v krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 16:26:33.050441 8277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41283 every 8 connection(s)
I20260929 16:26:33.050436 8147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41283
I20260929 16:26:33.051752 8147 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:26:33.052191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 8147
I20260929 16:26:33.052302 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:26:33.052603 392 external_mini_cluster.cc:1428] Setting key 2592eda2b72c69ac833fc11cf344f2d2
I20260929 16:26:33.052924 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.038973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53197 (local address 127.0.98.62:38557)
0929 16:26:33.039095 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:33.039104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:33.040200 (+ 1096us) server_negotiation.cc:407] Connection header received
0929 16:26:33.041417 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:33.041421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:33.041479 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:33.041562 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:33.042910 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.043493 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.044782 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.045009 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.045120 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.048549 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:33.048553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:33.048556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:33.048590 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:33.050513 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:33.051165 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:33.051168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:33.051169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:33.051254 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:33.051648 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:33.051650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:33.051651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:33.051869 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:33.051998 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:33.052636 (+ 638us) server_negotiation.cc:299] Negotiation successful
0929 16:26:33.052771 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 16:26:33.054013 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.039544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:53197)
0929 16:26:33.040049 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:33.040091 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:33.041171 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:33.041737 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:33.041744 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:33.042168 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:33.042746 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.042762 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.043623 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.043627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.044654 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.044664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.045160 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.045164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.045406 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.046142 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:33.046161 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:33.048355 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:33.050684 (+ 2329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:33.050690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:33.050706 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:33.051053 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:33.051358 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:33.051364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:33.051369 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:33.051550 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:33.051991 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:33.051998 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:33.052348 (+ 350us) client_negotiation.cc:769] Sending connection context
0929 16:26:33.052588 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 16:26:33.052953 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":147,"threads_started":1}
I20260929 16:26:33.056871 8278 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38557
I20260929 16:26:33.057055 8278 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:33.057668 8278 heartbeater.cc:511] Master 127.0.98.62:38557 requested a full tablet report, sending...
I20260929 16:26:33.058858 7934 ts_manager.cc:194] Registered new tserver with Master: 2ba27a37ecd048cf95328fc90b9b6565 (127.0.98.2:41283)
I20260929 16:26:33.059648 7934 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53197
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260929 16:26:33.066959 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.059157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36199 (local address 127.0.98.62:38557)
0929 16:26:33.059265 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:33.059269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:33.059541 (+ 272us) server_negotiation.cc:407] Connection header received
0929 16:26:33.059661 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:33.059667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:33.059731 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:33.059817 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:33.060313 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.060841 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.061742 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.061975 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.062080 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.063325 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:33.063332 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:33.063337 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:33.063369 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:33.064849 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:33.065453 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:33.065456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:33.065457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:33.065545 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:33.066034 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:33.066037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:33.066038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:33.066261 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:33.066377 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:33.066665 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 16:26:33.066778 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 16:26:33.066970 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.059112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.2:36199)
0929 16:26:33.059439 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:33.059454 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:33.059556 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:33.059849 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:33.059852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:33.060090 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:33.060201 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.060207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.060960 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.060964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.061629 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.061638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.062083 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.062087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.062175 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.062771 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:33.062783 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:33.063158 (+ 375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:33.065001 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:33.065005 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:33.065006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:33.065324 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:33.065703 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:33.065707 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:33.065710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:33.065842 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:33.066380 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:33.066385 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:33.066500 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 16:26:33.066610 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 16:26:33.066772 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":93,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 16:26:33.084775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38557
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:33.200255 8287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:33.200559 8287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:33.200639 8287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:33.205045 8287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:33.205152 8287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:33.205248 8287 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:26:33.205335 8287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:26:33.210983 8287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:33.212200 8287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:33.213305 8287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:26:33.223656 8287 server_base.cc:1034] running on GCE node
W20260929 16:26:33.223609 8292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:33.223591 8295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:33.223595 8293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:33.224484 8287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:33.225186 8287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:33.226469 8287 hybrid_clock.cc:648] HybridClock initialized: now 1790699193226453 us; error 32 us; skew 500 ppm
Sep 29 16:26:33 dist-test-slave-vq9v krb5kdc[7890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:33.229533 8287 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 16:26:33.230455 8287 webserver.cc:460] Webserver started at http://127.0.98.3:43039/ using document root <none> and password file <none>
I20260929 16:26:33.231009 8287 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:33.231070 8287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:33.231238 8287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:33.233421 8287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "a361147ca2454ab89c273e9bbd7fcdee"
format_stamp: "Formatted at 2026-09-29 16:26:33 on dist-test-slave-vq9v"
server_key: "5a02ef39997cd736d51d74cf1a456125"
server_key_iv: "73cb602e3cac618e896052207c5af41a"
server_key_version: "encryptionkey@0"
I20260929 16:26:33.234025 8287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a361147ca2454ab89c273e9bbd7fcdee"
format_stamp: "Formatted at 2026-09-29 16:26:33 on dist-test-slave-vq9v"
server_key: "5a02ef39997cd736d51d74cf1a456125"
server_key_iv: "73cb602e3cac618e896052207c5af41a"
server_key_version: "encryptionkey@0"
I20260929 16:26:33.238404 8287 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:26:33.241235 8302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:33.242713 8287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:26:33.242833 8287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "a361147ca2454ab89c273e9bbd7fcdee"
format_stamp: "Formatted at 2026-09-29 16:26:33 on dist-test-slave-vq9v"
server_key: "5a02ef39997cd736d51d74cf1a456125"
server_key_iv: "73cb602e3cac618e896052207c5af41a"
server_key_version: "encryptionkey@0"
I20260929 16:26:33.242944 8287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:33.270859 8287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:33.274941 8287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:33.275115 8287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:33.276736 8287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:33.276798 8287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:33.276847 8287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:33.276901 8287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:26:33 dist-test-slave-vq9v krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699193, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 16:26:33.287743 8287 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37993
I20260929 16:26:33.288033 8417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37993 every 8 connection(s)
I20260929 16:26:33.288733 8287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:26:33.290845 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.277629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55765 (local address 127.0.98.62:38557)
0929 16:26:33.277781 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:33.277787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:33.278711 (+ 924us) server_negotiation.cc:407] Connection header received
0929 16:26:33.279790 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:33.279793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:33.279855 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:33.279948 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:33.281759 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.282440 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.283735 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.283971 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.284084 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.286865 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:33.286869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:33.286872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:33.286901 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:33.288518 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:33.289070 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:33.289075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:33.289077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:33.289181 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:33.289702 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:33.289705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:33.289710 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:33.289961 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:33.290059 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:33.290567 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 16:26:33.290676 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:26:33.291604 8311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.278072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:55765)
0929 16:26:33.278533 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:33.278584 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:33.279499 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:33.280178 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:33.280190 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:33.280694 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:33.281512 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.281537 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.282569 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.282573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.283613 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.283623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.284133 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.284137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.284319 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.284954 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:33.284971 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:33.286687 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:33.288633 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:33.288639 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:33.288654 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:33.288972 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:33.289345 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:33.289348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:33.289350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:33.289519 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:33.290084 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:33.290090 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:33.290323 (+ 233us) client_negotiation.cc:769] Sending connection context
0929 16:26:33.290489 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 16:26:33.290742 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":135,"threads_started":1}
I20260929 16:26:33.291993 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 8287
I20260929 16:26:33.292128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:26:33.292472 392 external_mini_cluster.cc:1428] Setting key 7028c513b356fd1cff375ee5306f4b0f
I20260929 16:26:33.292999 8418 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38557
I20260929 16:26:33.293165 8418 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:33.293776 8418 heartbeater.cc:511] Master 127.0.98.62:38557 requested a full tablet report, sending...
I20260929 16:26:33.294858 7935 ts_manager.cc:194] Registered new tserver with Master: a361147ca2454ab89c273e9bbd7fcdee (127.0.98.3:37993)
I20260929 16:26:33.295648 7935 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55765
I20260929 16:26:33.302654 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.295601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37515 (local address 127.0.98.62:38557)
0929 16:26:33.295730 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:33.295734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:33.295944 (+ 210us) server_negotiation.cc:407] Connection header received
0929 16:26:33.296116 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:33.296119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:33.296165 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:33.296237 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:33.296712 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.297456 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.298243 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.298451 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.298529 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.299562 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:33.299565 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:33.299567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:33.299589 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:33.301000 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:33.301462 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:33.301469 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:33.301473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:33.301564 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:33.301845 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:33.301850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:33.301851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:33.302020 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:33.302125 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:33.302372 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 16:26:33.302494 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:26:33.302693 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.295539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.98.3:37515)
0929 16:26:33.295850 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:33.295867 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:33.296010 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:33.296326 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:33.296328 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:33.296503 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:33.296599 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.296605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.297582 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.297585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.298140 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.298148 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.298546 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.298549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.298615 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.299104 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:33.299115 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:33.299445 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:33.301104 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:33.301107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:33.301109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:33.301376 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:33.301658 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:33.301663 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:33.301665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:33.301756 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:33.302114 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:33.302117 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:33.302213 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 16:26:33.302318 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 16:26:33.302471 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":117,"threads_started":1}
I20260929 16:26:33.306795 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:26:33 dist-test-slave-vq9v krb5kdc[7890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699193, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:26:33 dist-test-slave-vq9v krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:33 dist-test-slave-vq9v krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:26:33.331410 7984 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:33.324993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53036 (local address 127.0.98.62:38557)
0929 16:26:33.325119 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:33.325122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:33.325136 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:26:33.325178 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:33.325180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:33.325265 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:33.325342 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:33.326133 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.326654 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.327523 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.327733 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.327811 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.331225 (+ 3414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53036: BlockingRecv error: recv got EOF from 127.0.0.1:53036 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 16:26:33 dist-test-slave-vq9v krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699193, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 16:26:33.341356 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.332801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53046 (local address 127.0.98.62:38557)
0929 16:26:33.332931 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:33.332936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:33.333127 (+ 191us) server_negotiation.cc:407] Connection header received
0929 16:26:33.333176 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:33.333178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:33.333313 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:33.333387 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:33.333963 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.334491 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.335278 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.335565 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.335646 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.338423 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:33.338426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:33.338428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:33.338453 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:33.339927 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:33.340280 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:33.340283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:33.340284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:33.340358 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:33.340580 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:33.340582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:33.340583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:33.340773 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:33.340874 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:33.341051 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 16:26:33.341136 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:26:33.344462 7935 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:26:33.347662 7935 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:26:33.362736 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.357265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41283 (local address 127.0.0.1:35316)
0929 16:26:33.357605 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:33.357644 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:33.357777 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:33.358483 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:33.358486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:33.358498 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:33.358671 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.358679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.360058 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.360061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.361349 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.361360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.362188 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.362191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.362295 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.362330 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:26:33.362450 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 16:26:33.362577 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":132,"spinlock_wait_cycles":4992,"thread_start_us":93,"threads_started":1}
I20260929 16:26:33.362744 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.357409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35316 (local address 127.0.98.2:41283)
0929 16:26:33.358082 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:33.358092 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:33.358109 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:26:33.358172 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:33.358176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:33.358341 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:33.358599 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:33.358768 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.359937 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.361502 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.362060 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.362187 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.362368 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:33.362477 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 16:26:33.362571 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":120,"threads_started":1}
I20260929 16:26:33.363013 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.356565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46171 (local address 127.0.0.1:39572)
0929 16:26:33.357571 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:33.357626 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:33.357776 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:33.358401 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:33.358410 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:33.358438 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:33.358603 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.358610 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.360094 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.360098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.361160 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.361168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.362492 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.362498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.362651 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.362726 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 16:26:33.362795 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 16:26:33.362873 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"mutex_wait_us":53,"spinlock_wait_cycles":10368,"thread_start_us":95,"threads_started":1}
I20260929 16:26:33.363086 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.356494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34910 (local address 127.0.98.3:37993)
0929 16:26:33.357310 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:33.357322 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:33.357786 (+ 464us) server_negotiation.cc:407] Connection header received
0929 16:26:33.358022 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:33.358029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:33.358244 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:33.358365 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:33.358888 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.359605 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.361476 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.362252 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.362344 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.362470 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:33.362857 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 16:26:33.362949 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":101,"threads_started":1}
I20260929 16:26:33.363436 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.356418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37993 (local address 127.0.0.1:34910)
0929 16:26:33.357692 (+ 1274us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:33.357706 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:33.357851 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:33.358555 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:33.358557 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:33.358596 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:33.358729 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.358737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.359720 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.359727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.361349 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.361360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.362515 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.362518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.362655 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.362721 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 16:26:33.363232 (+ 511us) client_negotiation.cc:240] Negotiation successful
0929 16:26:33.363311 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":224,"thread_start_us":91,"threads_started":1}
I20260929 16:26:33.363907 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.356732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39572 (local address 127.0.98.1:46171)
0929 16:26:33.357096 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:33.357105 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:33.357915 (+ 810us) server_negotiation.cc:407] Connection header received
0929 16:26:33.358082 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:33.358087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:33.358243 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:33.358377 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:33.358752 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.359937 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.361327 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.362150 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.362256 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.362425 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:33.363246 (+ 821us) server_negotiation.cc:299] Negotiation successful
0929 16:26:33.363771 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":69,"threads_started":1}
I20260929 16:26:33.366453 8072 tablet_service.cc:1467] Processing CreateTablet for tablet 9ff0ebf84787430c942d143debfbce55 (DEFAULT_TABLE table=test-table [id=322acc92a37d4dcc9e5bc86a1eb18189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:33.366468 8212 tablet_service.cc:1467] Processing CreateTablet for tablet 9ff0ebf84787430c942d143debfbce55 (DEFAULT_TABLE table=test-table [id=322acc92a37d4dcc9e5bc86a1eb18189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:33.366451 8352 tablet_service.cc:1467] Processing CreateTablet for tablet 9ff0ebf84787430c942d143debfbce55 (DEFAULT_TABLE table=test-table [id=322acc92a37d4dcc9e5bc86a1eb18189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:33.367614 8072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff0ebf84787430c942d143debfbce55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:33.367614 8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff0ebf84787430c942d143debfbce55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:33.367623 8352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff0ebf84787430c942d143debfbce55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:33.374956 8446 tablet_bootstrap.cc:492] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee: Bootstrap starting.
I20260929 16:26:33.376425 8447 tablet_bootstrap.cc:492] T 9ff0ebf84787430c942d143debfbce55 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Bootstrap starting.
I20260929 16:26:33.377509 8446 tablet_bootstrap.cc:654] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:33.377592 8448 tablet_bootstrap.cc:492] T 9ff0ebf84787430c942d143debfbce55 P 2ba27a37ecd048cf95328fc90b9b6565: Bootstrap starting.
I20260929 16:26:33.378310 8446 log.cc:824] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:33.378742 8447 tablet_bootstrap.cc:654] T 9ff0ebf84787430c942d143debfbce55 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:33.379614 8447 log.cc:824] T 9ff0ebf84787430c942d143debfbce55 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:33.379889 8448 tablet_bootstrap.cc:654] T 9ff0ebf84787430c942d143debfbce55 P 2ba27a37ecd048cf95328fc90b9b6565: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:33.380741 8448 log.cc:824] T 9ff0ebf84787430c942d143debfbce55 P 2ba27a37ecd048cf95328fc90b9b6565: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:33.381067 8446 tablet_bootstrap.cc:492] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee: No bootstrap required, opened a new log
I20260929 16:26:33.381331 8446 ts_tablet_manager.cc:1397] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 16:26:33.381822 8447 tablet_bootstrap.cc:492] T 9ff0ebf84787430c942d143debfbce55 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: No bootstrap required, opened a new log
I20260929 16:26:33.382019 8447 ts_tablet_manager.cc:1397] T 9ff0ebf84787430c942d143debfbce55 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:26:33.383225 8448 tablet_bootstrap.cc:492] T 9ff0ebf84787430c942d143debfbce55 P 2ba27a37ecd048cf95328fc90b9b6565: No bootstrap required, opened a new log
I20260929 16:26:33.383458 8448 ts_tablet_manager.cc:1397] T 9ff0ebf84787430c942d143debfbce55 P 2ba27a37ecd048cf95328fc90b9b6565: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:26:33.384337 8446 raft_consensus.cc:348] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } } peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } }
I20260929 16:26:33.384618 8446 raft_consensus.cc:374] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:33.384678 8446 raft_consensus.cc:729] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a361147ca2454ab89c273e9bbd7fcdee, State: Initialized, Role: FOLLOWER
I20260929 16:26:33.385142 8446 consensus_queue.cc:260] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } } peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } }
I20260929 16:26:33.385910 8418 heartbeater.cc:503] Master 127.0.98.62:38557 was elected leader, sending a full tablet report...
I20260929 16:26:33.386313 8446 ts_tablet_manager.cc:1428] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:26:33.386407 8447 raft_consensus.cc:348] T 9ff0ebf84787430c942d143debfbce55 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } } peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } }
I20260929 16:26:33.386669 8447 raft_consensus.cc:374] T 9ff0ebf84787430c942d143debfbce55 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:33.386759 8447 raft_consensus.cc:729] T 9ff0ebf84787430c942d143debfbce55 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c12d4976bc4aa3a67d4ff1a5ab7a57, State: Initialized, Role: FOLLOWER
I20260929 16:26:33.387282 8447 consensus_queue.cc:260] T 9ff0ebf84787430c942d143debfbce55 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } } peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } }
I20260929 16:26:33.387355 8448 raft_consensus.cc:348] T 9ff0ebf84787430c942d143debfbce55 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } } peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } }
I20260929 16:26:33.387601 8448 raft_consensus.cc:374] T 9ff0ebf84787430c942d143debfbce55 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:33.387651 8448 raft_consensus.cc:729] T 9ff0ebf84787430c942d143debfbce55 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba27a37ecd048cf95328fc90b9b6565, State: Initialized, Role: FOLLOWER
I20260929 16:26:33.388100 8448 consensus_queue.cc:260] T 9ff0ebf84787430c942d143debfbce55 P 2ba27a37ecd048cf95328fc90b9b6565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } } peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } }
I20260929 16:26:33.388275 8447 ts_tablet_manager.cc:1428] T 9ff0ebf84787430c942d143debfbce55 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:26:33.388594 8138 heartbeater.cc:503] Master 127.0.98.62:38557 was elected leader, sending a full tablet report...
I20260929 16:26:33.388938 8278 heartbeater.cc:503] Master 127.0.98.62:38557 was elected leader, sending a full tablet report...
I20260929 16:26:33.389020 8448 ts_tablet_manager.cc:1428] T 9ff0ebf84787430c942d143debfbce55 P 2ba27a37ecd048cf95328fc90b9b6565: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260929 16:26:33.539896 8419 tablet.cc:2369] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:26:33.550063 8452 raft_consensus.cc:482] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:26:33.550282 8452 raft_consensus.cc:504] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } } peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } }
I20260929 16:26:33.551496 8452 leader_election.cc:288] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3c12d4976bc4aa3a67d4ff1a5ab7a57 (127.0.98.1:46171), 2ba27a37ecd048cf95328fc90b9b6565 (127.0.98.2:41283)
W20260929 16:26:33.553323 8279 tablet.cc:2369] T 9ff0ebf84787430c942d143debfbce55 P 2ba27a37ecd048cf95328fc90b9b6565: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:26:33.555994 8311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.551740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46171 (local address 127.0.98.3:39383)
0929 16:26:33.551894 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:33.551928 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:33.552036 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:33.552292 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:33.552294 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:33.552309 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:33.552503 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.552512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.553626 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.553629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.554638 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.554652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.555487 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.555492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.555606 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.555636 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:26:33.555737 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 16:26:33.555821 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"spinlock_wait_cycles":13440}
I20260929 16:26:33.556008 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.551880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39383 (local address 127.0.98.1:46171)
0929 16:26:33.552030 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:33.552034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:33.552051 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:26:33.552143 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:33.552146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:33.552200 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:33.552282 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:33.552651 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.553501 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.554784 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.555358 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.555438 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.555489 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:33.555772 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 16:26:33.555844 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:26:33.556493 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.552091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38467 (local address 127.0.98.2:41283)
0929 16:26:33.552226 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:33.552229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:33.552364 (+ 135us) server_negotiation.cc:407] Connection header received
0929 16:26:33.552500 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:33.552503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:33.552567 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:33.552651 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:33.553531 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.554042 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.555227 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.555808 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.555917 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.555979 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:33.556236 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 16:26:33.556324 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:26:33.556843 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.551802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41283 (local address 127.0.98.3:38467)
0929 16:26:33.552263 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:33.552278 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:33.552390 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:33.552668 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:33.552671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:33.552688 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:33.553388 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.553399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.554154 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.554156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.555061 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.555078 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.555942 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.555946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.556060 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.556094 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:26:33.556669 (+ 575us) client_negotiation.cc:240] Negotiation successful
0929 16:26:33.556738 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"spinlock_wait_cycles":896,"thread_start_us":142,"threads_started":1}
I20260929 16:26:33.556802 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff0ebf84787430c942d143debfbce55" candidate_uuid: "a361147ca2454ab89c273e9bbd7fcdee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" is_pre_election: true
I20260929 16:26:33.557183 8092 raft_consensus.cc:2454] T 9ff0ebf84787430c942d143debfbce55 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a361147ca2454ab89c273e9bbd7fcdee in term 0.
I20260929 16:26:33.557514 8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff0ebf84787430c942d143debfbce55" candidate_uuid: "a361147ca2454ab89c273e9bbd7fcdee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ba27a37ecd048cf95328fc90b9b6565" is_pre_election: true
I20260929 16:26:33.557811 8232 raft_consensus.cc:2454] T 9ff0ebf84787430c942d143debfbce55 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a361147ca2454ab89c273e9bbd7fcdee in term 0.
I20260929 16:26:33.557781 8305 leader_election.cc:302] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a361147ca2454ab89c273e9bbd7fcdee, a3c12d4976bc4aa3a67d4ff1a5ab7a57; no voters:
I20260929 16:26:33.558092 8452 raft_consensus.cc:2781] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:26:33.558205 8452 raft_consensus.cc:482] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:26:33.558257 8452 raft_consensus.cc:3037] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:33.559284 8452 raft_consensus.cc:504] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } } peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } }
I20260929 16:26:33.559654 8452 leader_election.cc:288] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee [CANDIDATE]: Term 1 election: Requested vote from peers a3c12d4976bc4aa3a67d4ff1a5ab7a57 (127.0.98.1:46171), 2ba27a37ecd048cf95328fc90b9b6565 (127.0.98.2:41283)
I20260929 16:26:33.560074 8232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff0ebf84787430c942d143debfbce55" candidate_uuid: "a361147ca2454ab89c273e9bbd7fcdee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ba27a37ecd048cf95328fc90b9b6565"
I20260929 16:26:33.560077 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff0ebf84787430c942d143debfbce55" candidate_uuid: "a361147ca2454ab89c273e9bbd7fcdee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57"
I20260929 16:26:33.560210 8232 raft_consensus.cc:3037] T 9ff0ebf84787430c942d143debfbce55 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:33.560210 8092 raft_consensus.cc:3037] T 9ff0ebf84787430c942d143debfbce55 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:33.561574 8092 raft_consensus.cc:2454] T 9ff0ebf84787430c942d143debfbce55 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a361147ca2454ab89c273e9bbd7fcdee in term 1.
I20260929 16:26:33.561574 8232 raft_consensus.cc:2454] T 9ff0ebf84787430c942d143debfbce55 P 2ba27a37ecd048cf95328fc90b9b6565 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a361147ca2454ab89c273e9bbd7fcdee in term 1.
I20260929 16:26:33.561909 8304 leader_election.cc:302] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba27a37ecd048cf95328fc90b9b6565, a361147ca2454ab89c273e9bbd7fcdee; no voters:
I20260929 16:26:33.562125 8452 raft_consensus.cc:2781] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:33.562302 8452 raft_consensus.cc:686] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee [term 1 LEADER]: Becoming Leader. State: Replica: a361147ca2454ab89c273e9bbd7fcdee, State: Running, Role: LEADER
I20260929 16:26:33.562638 8452 consensus_queue.cc:237] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } } peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } }
I20260929 16:26:33.566022 7933 catalog_manager.cc:5495] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee reported cstate change: term changed from 0 to 1, leader changed from <none> to a361147ca2454ab89c273e9bbd7fcdee (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a361147ca2454ab89c273e9bbd7fcdee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } health_report { overall_health: UNKNOWN } } }
W20260929 16:26:33.581297 8139 tablet.cc:2369] T 9ff0ebf84787430c942d143debfbce55 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:26:33.601464 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.597819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.0.98.3:37993)
0929 16:26:33.597964 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:33.597971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:33.597985 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:26:33.598165 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:33.598168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:33.598223 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:33.598329 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:33.598580 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.599090 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.599945 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.600162 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.600269 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.600560 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:33.601007 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:33.601137 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:33.601226 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 16:26:33.601288 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:26:33.607777 8232 raft_consensus.cc:1261] T 9ff0ebf84787430c942d143debfbce55 P 2ba27a37ecd048cf95328fc90b9b6565 [term 1 FOLLOWER]: Refusing update from remote peer a361147ca2454ab89c273e9bbd7fcdee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:33.607774 8092 raft_consensus.cc:1261] T 9ff0ebf84787430c942d143debfbce55 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 1 FOLLOWER]: Refusing update from remote peer a361147ca2454ab89c273e9bbd7fcdee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:33.608543 8452 consensus_queue.cc:1035] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:33.608800 8456 consensus_queue.cc:1035] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:33.617326 8461 mvcc.cc:205] Tried to move back new op lower bound from 7334703897009283072 to 7334703896835231744. Current Snapshot: MvccSnapshot[applied={T|T < 7334703897009283072}]
I20260929 16:26:33.628743 7935 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:33.628955 7935 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:33.632191 7935 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b: Sending DeleteTablet for 3 replicas of tablet 9ff0ebf84787430c942d143debfbce55
I20260929 16:26:33.633066 8352 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ff0ebf84787430c942d143debfbce55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34910
I20260929 16:26:33.633380 8212 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ff0ebf84787430c942d143debfbce55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35316
I20260929 16:26:33.633385 8072 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ff0ebf84787430c942d143debfbce55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39572
I20260929 16:26:33.633791 8469 tablet_replica.cc:330] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee: stopping tablet replica
I20260929 16:26:33.633903 8471 tablet_replica.cc:330] T 9ff0ebf84787430c942d143debfbce55 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: stopping tablet replica
I20260929 16:26:33.634222 8471 raft_consensus.cc:2229] T 9ff0ebf84787430c942d143debfbce55 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:33.634223 8469 raft_consensus.cc:2229] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:26:33.634570 8471 raft_consensus.cc:2258] T 9ff0ebf84787430c942d143debfbce55 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:33.634807 8469 raft_consensus.cc:2258] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:33.634850 7933 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:26:33.635319 7933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:26:33.636375 8470 tablet_replica.cc:330] T 9ff0ebf84787430c942d143debfbce55 P 2ba27a37ecd048cf95328fc90b9b6565: stopping tablet replica
I20260929 16:26:33.636722 8470 raft_consensus.cc:2229] T 9ff0ebf84787430c942d143debfbce55 P 2ba27a37ecd048cf95328fc90b9b6565 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:33.637099 8470 raft_consensus.cc:2258] T 9ff0ebf84787430c942d143debfbce55 P 2ba27a37ecd048cf95328fc90b9b6565 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:33.637511 8469 ts_tablet_manager.cc:1905] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:33.637512 8471 ts_tablet_manager.cc:1905] T 9ff0ebf84787430c942d143debfbce55 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:33.638460 8470 ts_tablet_manager.cc:1905] T 9ff0ebf84787430c942d143debfbce55 P 2ba27a37ecd048cf95328fc90b9b6565: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:33.640417 8471 ts_tablet_manager.cc:1918] T 9ff0ebf84787430c942d143debfbce55 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:26:33.640518 8471 log.cc:1196] T 9ff0ebf84787430c942d143debfbce55 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-0/wal/wals/9ff0ebf84787430c942d143debfbce55
I20260929 16:26:33.640868 8471 ts_tablet_manager.cc:1939] T 9ff0ebf84787430c942d143debfbce55 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Deleting consensus metadata
I20260929 16:26:33.640951 8469 ts_tablet_manager.cc:1918] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:26:33.640951 8470 ts_tablet_manager.cc:1918] T 9ff0ebf84787430c942d143debfbce55 P 2ba27a37ecd048cf95328fc90b9b6565: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:26:33.641050 8469 log.cc:1196] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/9ff0ebf84787430c942d143debfbce55
I20260929 16:26:33.641050 8470 log.cc:1196] T 9ff0ebf84787430c942d143debfbce55 P 2ba27a37ecd048cf95328fc90b9b6565: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/9ff0ebf84787430c942d143debfbce55
I20260929 16:26:33.641390 8470 ts_tablet_manager.cc:1939] T 9ff0ebf84787430c942d143debfbce55 P 2ba27a37ecd048cf95328fc90b9b6565: Deleting consensus metadata
I20260929 16:26:33.641407 8469 ts_tablet_manager.cc:1939] T 9ff0ebf84787430c942d143debfbce55 P a361147ca2454ab89c273e9bbd7fcdee: Deleting consensus metadata
I20260929 16:26:33.642304 7921 catalog_manager.cc:4841] TS a3c12d4976bc4aa3a67d4ff1a5ab7a57 (127.0.98.1:46171): tablet 9ff0ebf84787430c942d143debfbce55 (table test-table [id=322acc92a37d4dcc9e5bc86a1eb18189]) successfully deleted
I20260929 16:26:33.642452 7920 catalog_manager.cc:4841] TS 2ba27a37ecd048cf95328fc90b9b6565 (127.0.98.2:41283): tablet 9ff0ebf84787430c942d143debfbce55 (table test-table [id=322acc92a37d4dcc9e5bc86a1eb18189]) successfully deleted
I20260929 16:26:33.642589 7920 catalog_manager.cc:4841] TS a361147ca2454ab89c273e9bbd7fcdee (127.0.98.3:37993): tablet 9ff0ebf84787430c942d143debfbce55 (table test-table [id=322acc92a37d4dcc9e5bc86a1eb18189]) successfully deleted
I20260929 16:26:33.643491 8072 tablet_service.cc:1467] Processing CreateTablet for tablet 209f9dfe06534bfcafa9b1122c6bcf17 (DEFAULT_TABLE table=test-table [id=97db24d93a77492ab4cecc9a3c618d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:33.643559 8212 tablet_service.cc:1467] Processing CreateTablet for tablet 209f9dfe06534bfcafa9b1122c6bcf17 (DEFAULT_TABLE table=test-table [id=97db24d93a77492ab4cecc9a3c618d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:33.643787 8072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209f9dfe06534bfcafa9b1122c6bcf17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:33.643834 8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209f9dfe06534bfcafa9b1122c6bcf17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:33.645933 8447 tablet_bootstrap.cc:492] T 209f9dfe06534bfcafa9b1122c6bcf17 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Bootstrap starting.
I20260929 16:26:33.646236 8448 tablet_bootstrap.cc:492] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565: Bootstrap starting.
I20260929 16:26:33.646708 8352 tablet_service.cc:1467] Processing CreateTablet for tablet 209f9dfe06534bfcafa9b1122c6bcf17 (DEFAULT_TABLE table=test-table [id=97db24d93a77492ab4cecc9a3c618d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:33.646976 8352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209f9dfe06534bfcafa9b1122c6bcf17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:33.646992 8447 tablet_bootstrap.cc:654] T 209f9dfe06534bfcafa9b1122c6bcf17 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:33.647142 8448 tablet_bootstrap.cc:654] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:33.648756 8448 tablet_bootstrap.cc:492] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565: No bootstrap required, opened a new log
I20260929 16:26:33.648828 8447 tablet_bootstrap.cc:492] T 209f9dfe06534bfcafa9b1122c6bcf17 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: No bootstrap required, opened a new log
I20260929 16:26:33.648844 8448 ts_tablet_manager.cc:1397] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:26:33.648936 8447 ts_tablet_manager.cc:1397] T 209f9dfe06534bfcafa9b1122c6bcf17 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:26:33.649394 8446 tablet_bootstrap.cc:492] T 209f9dfe06534bfcafa9b1122c6bcf17 P a361147ca2454ab89c273e9bbd7fcdee: Bootstrap starting.
I20260929 16:26:33.649521 8448 raft_consensus.cc:348] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:33.649633 8448 raft_consensus.cc:374] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:33.649573 8447 raft_consensus.cc:348] T 209f9dfe06534bfcafa9b1122c6bcf17 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:33.649669 8448 raft_consensus.cc:729] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba27a37ecd048cf95328fc90b9b6565, State: Initialized, Role: FOLLOWER
I20260929 16:26:33.649678 8447 raft_consensus.cc:374] T 209f9dfe06534bfcafa9b1122c6bcf17 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:33.649726 8447 raft_consensus.cc:729] T 209f9dfe06534bfcafa9b1122c6bcf17 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c12d4976bc4aa3a67d4ff1a5ab7a57, State: Initialized, Role: FOLLOWER
I20260929 16:26:33.649808 8448 consensus_queue.cc:260] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:33.649855 8447 consensus_queue.cc:260] T 209f9dfe06534bfcafa9b1122c6bcf17 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:33.650089 8448 ts_tablet_manager.cc:1428] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:26:33.650102 8447 ts_tablet_manager.cc:1428] T 209f9dfe06534bfcafa9b1122c6bcf17 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 16:26:33.650389 8446 tablet_bootstrap.cc:654] T 209f9dfe06534bfcafa9b1122c6bcf17 P a361147ca2454ab89c273e9bbd7fcdee: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:33.651582 8446 tablet_bootstrap.cc:492] T 209f9dfe06534bfcafa9b1122c6bcf17 P a361147ca2454ab89c273e9bbd7fcdee: No bootstrap required, opened a new log
I20260929 16:26:33.651693 8446 ts_tablet_manager.cc:1397] T 209f9dfe06534bfcafa9b1122c6bcf17 P a361147ca2454ab89c273e9bbd7fcdee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:26:33.652084 8446 raft_consensus.cc:348] T 209f9dfe06534bfcafa9b1122c6bcf17 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:33.652232 8446 raft_consensus.cc:374] T 209f9dfe06534bfcafa9b1122c6bcf17 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:33.652276 8446 raft_consensus.cc:729] T 209f9dfe06534bfcafa9b1122c6bcf17 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a361147ca2454ab89c273e9bbd7fcdee, State: Initialized, Role: FOLLOWER
I20260929 16:26:33.652406 8446 consensus_queue.cc:260] T 209f9dfe06534bfcafa9b1122c6bcf17 P a361147ca2454ab89c273e9bbd7fcdee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:33.652680 8446 ts_tablet_manager.cc:1428] T 209f9dfe06534bfcafa9b1122c6bcf17 P a361147ca2454ab89c273e9bbd7fcdee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:26:33.751924 8454 raft_consensus.cc:482] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:26:33.752099 8454 raft_consensus.cc:504] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:33.753322 8454 leader_election.cc:288] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3c12d4976bc4aa3a67d4ff1a5ab7a57 (127.0.98.1:46171), a361147ca2454ab89c273e9bbd7fcdee (127.0.98.3:37993)
I20260929 16:26:33.758126 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.754045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39275 (local address 127.0.98.3:37993)
0929 16:26:33.754162 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:33.754166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:33.754181 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:26:33.754235 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:33.754238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:33.754285 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:33.754376 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:33.754761 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.755496 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.756994 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.757493 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.757612 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.757683 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:33.757885 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 16:26:33.757967 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:26:33.758248 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.753508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46171 (local address 127.0.98.2:54025)
0929 16:26:33.753857 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:33.753874 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:33.754040 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:33.754460 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:33.754463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:33.754474 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:33.754637 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.754644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.755376 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.755378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.756793 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.756805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.757804 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.757810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.757913 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.757934 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:26:33.758010 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 16:26:33.758081 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":67,"thread_start_us":102,"threads_started":1}
I20260929 16:26:33.758265 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.753692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54025 (local address 127.0.98.1:46171)
0929 16:26:33.753842 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:33.753847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:33.753955 (+ 108us) server_negotiation.cc:407] Connection header received
0929 16:26:33.754190 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:33.754195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:33.754251 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:33.754309 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:33.754760 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.755251 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.756979 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.757660 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.757813 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.757878 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:33.758057 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 16:26:33.758135 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:26:33.758739 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209f9dfe06534bfcafa9b1122c6bcf17" candidate_uuid: "2ba27a37ecd048cf95328fc90b9b6565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" is_pre_election: true
I20260929 16:26:33.758910 8092 raft_consensus.cc:2454] T 209f9dfe06534bfcafa9b1122c6bcf17 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ba27a37ecd048cf95328fc90b9b6565 in term 0.
I20260929 16:26:33.758981 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.753631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37993 (local address 127.0.98.2:39275)
0929 16:26:33.754043 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:33.754055 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:33.754130 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:33.754403 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:33.754408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:33.754429 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:33.754609 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.754618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.755615 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.755618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.756793 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.756805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.757621 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.757625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.757726 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.757749 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:26:33.758768 (+ 1019us) client_negotiation.cc:240] Negotiation successful
0929 16:26:33.758842 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":28,"thread_start_us":126,"threads_started":1}
I20260929 16:26:33.759352 8165 leader_election.cc:302] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba27a37ecd048cf95328fc90b9b6565, a3c12d4976bc4aa3a67d4ff1a5ab7a57; no voters:
I20260929 16:26:33.759660 8454 raft_consensus.cc:2781] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:26:33.759629 8372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209f9dfe06534bfcafa9b1122c6bcf17" candidate_uuid: "2ba27a37ecd048cf95328fc90b9b6565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a361147ca2454ab89c273e9bbd7fcdee" is_pre_election: true
I20260929 16:26:33.759749 8454 raft_consensus.cc:482] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:26:33.759796 8454 raft_consensus.cc:3037] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:33.759876 8372 raft_consensus.cc:2454] T 209f9dfe06534bfcafa9b1122c6bcf17 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ba27a37ecd048cf95328fc90b9b6565 in term 0.
I20260929 16:26:33.760836 8454 raft_consensus.cc:504] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:33.761193 8454 leader_election.cc:288] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565 [CANDIDATE]: Term 1 election: Requested vote from peers a3c12d4976bc4aa3a67d4ff1a5ab7a57 (127.0.98.1:46171), a361147ca2454ab89c273e9bbd7fcdee (127.0.98.3:37993)
I20260929 16:26:33.761541 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209f9dfe06534bfcafa9b1122c6bcf17" candidate_uuid: "2ba27a37ecd048cf95328fc90b9b6565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57"
I20260929 16:26:33.761639 8372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209f9dfe06534bfcafa9b1122c6bcf17" candidate_uuid: "2ba27a37ecd048cf95328fc90b9b6565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a361147ca2454ab89c273e9bbd7fcdee"
I20260929 16:26:33.761680 8092 raft_consensus.cc:3037] T 209f9dfe06534bfcafa9b1122c6bcf17 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:33.761761 8372 raft_consensus.cc:3037] T 209f9dfe06534bfcafa9b1122c6bcf17 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:33.762887 8372 raft_consensus.cc:2454] T 209f9dfe06534bfcafa9b1122c6bcf17 P a361147ca2454ab89c273e9bbd7fcdee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ba27a37ecd048cf95328fc90b9b6565 in term 1.
I20260929 16:26:33.762885 8092 raft_consensus.cc:2454] T 209f9dfe06534bfcafa9b1122c6bcf17 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ba27a37ecd048cf95328fc90b9b6565 in term 1.
I20260929 16:26:33.763165 8165 leader_election.cc:302] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba27a37ecd048cf95328fc90b9b6565, a3c12d4976bc4aa3a67d4ff1a5ab7a57; no voters:
I20260929 16:26:33.763358 8454 raft_consensus.cc:2781] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:33.763617 8454 raft_consensus.cc:686] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565 [term 1 LEADER]: Becoming Leader. State: Replica: 2ba27a37ecd048cf95328fc90b9b6565, State: Running, Role: LEADER
I20260929 16:26:33.763871 8454 consensus_queue.cc:237] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:33.766566 7933 catalog_manager.cc:5495] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ba27a37ecd048cf95328fc90b9b6565 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2ba27a37ecd048cf95328fc90b9b6565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } health_report { overall_health: UNKNOWN } } }
Sep 29 16:26:33 dist-test-slave-vq9v krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699192, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 16:26:33.807646 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.799027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38557 (local address 127.0.0.1:53060)
0929 16:26:33.799399 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:33.799413 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:33.799549 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:33.799829 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:33.799832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:33.800244 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:33.800338 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.800345 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.801078 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.801084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.801703 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.801713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.802130 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.802133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.802211 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.802805 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:33.802817 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:33.804471 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:33.806182 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:33.806190 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:33.806195 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:33.806441 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:33.806677 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:33.806680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:33.806681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:33.806754 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:33.807170 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:33.807176 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:33.807277 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 16:26:33.807389 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 16:26:33.807512 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":106,"threads_started":1}
I20260929 16:26:33.807760 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.799113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53060 (local address 127.0.98.62:38557)
0929 16:26:33.799285 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:33.799291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:33.799491 (+ 200us) server_negotiation.cc:407] Connection header received
0929 16:26:33.799657 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:33.799659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:33.799722 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:33.799809 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:33.800455 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.800973 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.801813 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.802023 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.802128 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.804626 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:33.804633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:33.804635 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:33.804657 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:33.806083 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:33.806529 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:33.806532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:33.806533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:33.806603 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:33.806858 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:33.806861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:33.806862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:33.807064 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:33.807166 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:33.807444 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 16:26:33.807574 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":77}
I20260929 16:26:33.811694 7933 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53060:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 16:26:33.812358 7933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:26:33.817469 8072 tablet_service.cc:1467] Processing CreateTablet for tablet 6bad9d10be1a42f08f4ca81a8fcb1d87 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=624f0c36b15e4b04ab51acd380cabe70]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:26:33.817469 8212 tablet_service.cc:1467] Processing CreateTablet for tablet 6bad9d10be1a42f08f4ca81a8fcb1d87 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=624f0c36b15e4b04ab51acd380cabe70]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:26:33.817668 8352 tablet_service.cc:1467] Processing CreateTablet for tablet 6bad9d10be1a42f08f4ca81a8fcb1d87 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=624f0c36b15e4b04ab51acd380cabe70]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:26:33.817868 8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bad9d10be1a42f08f4ca81a8fcb1d87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:33.817863 8072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bad9d10be1a42f08f4ca81a8fcb1d87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:33.818054 8352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bad9d10be1a42f08f4ca81a8fcb1d87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:33.820533 8446 tablet_bootstrap.cc:492] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a361147ca2454ab89c273e9bbd7fcdee: Bootstrap starting.
I20260929 16:26:33.820686 8448 tablet_bootstrap.cc:492] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565: Bootstrap starting.
I20260929 16:26:33.821420 8446 tablet_bootstrap.cc:654] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a361147ca2454ab89c273e9bbd7fcdee: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:33.821570 8448 tablet_bootstrap.cc:654] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:33.822400 8447 tablet_bootstrap.cc:492] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Bootstrap starting.
I20260929 16:26:33.822615 8446 tablet_bootstrap.cc:492] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a361147ca2454ab89c273e9bbd7fcdee: No bootstrap required, opened a new log
I20260929 16:26:33.822692 8448 tablet_bootstrap.cc:492] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565: No bootstrap required, opened a new log
I20260929 16:26:33.822691 8446 ts_tablet_manager.cc:1397] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a361147ca2454ab89c273e9bbd7fcdee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:26:33.822757 8448 ts_tablet_manager.cc:1397] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:26:33.823099 8446 raft_consensus.cc:348] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:33.823182 8447 tablet_bootstrap.cc:654] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:33.823137 8448 raft_consensus.cc:348] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:33.823215 8446 raft_consensus.cc:374] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:33.823226 8448 raft_consensus.cc:374] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:33.823253 8446 raft_consensus.cc:729] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a361147ca2454ab89c273e9bbd7fcdee, State: Initialized, Role: FOLLOWER
I20260929 16:26:33.823261 8448 raft_consensus.cc:729] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba27a37ecd048cf95328fc90b9b6565, State: Initialized, Role: FOLLOWER
I20260929 16:26:33.823395 8446 consensus_queue.cc:260] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a361147ca2454ab89c273e9bbd7fcdee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:33.823402 8448 consensus_queue.cc:260] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:33.823673 8446 ts_tablet_manager.cc:1428] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a361147ca2454ab89c273e9bbd7fcdee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:26:33.823671 8448 ts_tablet_manager.cc:1428] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 16:26:33.823649 8464 tablet_replica.cc:439] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a361147ca2454ab89c273e9bbd7fcdee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } } }
I20260929 16:26:33.823800 8464 tablet_replica.cc:442] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a361147ca2454ab89c273e9bbd7fcdee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:26:33.823974 8454 tablet_replica.cc:439] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } } }
I20260929 16:26:33.824093 8454 tablet_replica.cc:442] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:26:33.825143 8447 tablet_bootstrap.cc:492] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: No bootstrap required, opened a new log
I20260929 16:26:33.825264 8447 ts_tablet_manager.cc:1397] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:26:33.825763 8447 raft_consensus.cc:348] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:33.825883 8447 raft_consensus.cc:374] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:33.825924 8447 raft_consensus.cc:729] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c12d4976bc4aa3a67d4ff1a5ab7a57, State: Initialized, Role: FOLLOWER
I20260929 16:26:33.826100 8447 consensus_queue.cc:260] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:33.826396 8447 ts_tablet_manager.cc:1428] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:26:33.826629 8453 tablet_replica.cc:439] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } } }
I20260929 16:26:33.826766 8453 tablet_replica.cc:442] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:26:33.896952 8454 raft_consensus.cc:482] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:26:33.897106 8454 raft_consensus.cc:504] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:33.897626 8454 leader_election.cc:288] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3c12d4976bc4aa3a67d4ff1a5ab7a57 (127.0.98.1:46171), a361147ca2454ab89c273e9bbd7fcdee (127.0.98.3:37993)
I20260929 16:26:33.898062 8372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bad9d10be1a42f08f4ca81a8fcb1d87" candidate_uuid: "2ba27a37ecd048cf95328fc90b9b6565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a361147ca2454ab89c273e9bbd7fcdee" is_pre_election: true
I20260929 16:26:33.898070 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bad9d10be1a42f08f4ca81a8fcb1d87" candidate_uuid: "2ba27a37ecd048cf95328fc90b9b6565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" is_pre_election: true
I20260929 16:26:33.898221 8372 raft_consensus.cc:2454] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ba27a37ecd048cf95328fc90b9b6565 in term 0.
I20260929 16:26:33.898233 8092 raft_consensus.cc:2454] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ba27a37ecd048cf95328fc90b9b6565 in term 0.
I20260929 16:26:33.898532 8165 leader_election.cc:302] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba27a37ecd048cf95328fc90b9b6565, a3c12d4976bc4aa3a67d4ff1a5ab7a57; no voters:
I20260929 16:26:33.898710 8454 raft_consensus.cc:2781] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:26:33.898772 8454 raft_consensus.cc:482] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:26:33.898838 8454 raft_consensus.cc:3037] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:33.899850 8454 raft_consensus.cc:504] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:33.900209 8454 leader_election.cc:288] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565 [CANDIDATE]: Term 1 election: Requested vote from peers a3c12d4976bc4aa3a67d4ff1a5ab7a57 (127.0.98.1:46171), a361147ca2454ab89c273e9bbd7fcdee (127.0.98.3:37993)
I20260929 16:26:33.900682 8372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bad9d10be1a42f08f4ca81a8fcb1d87" candidate_uuid: "2ba27a37ecd048cf95328fc90b9b6565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a361147ca2454ab89c273e9bbd7fcdee"
I20260929 16:26:33.900719 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bad9d10be1a42f08f4ca81a8fcb1d87" candidate_uuid: "2ba27a37ecd048cf95328fc90b9b6565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57"
I20260929 16:26:33.900847 8372 raft_consensus.cc:3037] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:33.900914 8092 raft_consensus.cc:3037] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:33.901908 8372 raft_consensus.cc:2454] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a361147ca2454ab89c273e9bbd7fcdee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ba27a37ecd048cf95328fc90b9b6565 in term 1.
I20260929 16:26:33.901957 8092 raft_consensus.cc:2454] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ba27a37ecd048cf95328fc90b9b6565 in term 1.
I20260929 16:26:33.902244 8164 leader_election.cc:302] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba27a37ecd048cf95328fc90b9b6565, a361147ca2454ab89c273e9bbd7fcdee; no voters:
I20260929 16:26:33.902441 8454 raft_consensus.cc:2781] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:33.902532 8454 raft_consensus.cc:686] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565 [term 1 LEADER]: Becoming Leader. State: Replica: 2ba27a37ecd048cf95328fc90b9b6565, State: Running, Role: LEADER
I20260929 16:26:33.902676 8454 consensus_queue.cc:237] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:33.903489 8477 tablet_replica.cc:439] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565: TxnStatusTablet state changed. Reason: New leader 2ba27a37ecd048cf95328fc90b9b6565. Latest consensus state: current_term: 1 leader_uuid: "2ba27a37ecd048cf95328fc90b9b6565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } } }
I20260929 16:26:33.903627 8477 tablet_replica.cc:442] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:26:33.903927 8489 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 16:26:33.904116 8489 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 16:26:33.904822 7935 catalog_manager.cc:5495] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ba27a37ecd048cf95328fc90b9b6565 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2ba27a37ecd048cf95328fc90b9b6565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } health_report { overall_health: UNKNOWN } } }
I20260929 16:26:33.926913 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.922608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41283 (local address 127.0.0.1:35320)
0929 16:26:33.922768 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:33.922788 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:33.922911 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:33.923258 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:33.923262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:33.923273 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:26:33.923397 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.923403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.924221 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.924228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.924932 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.924941 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.925636 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.925639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.925726 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.925786 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:26:33.926596 (+ 810us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:26:33.926611 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 16:26:33.926685 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 16:26:33.926778 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 16:26:33.926918 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.922755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35320 (local address 127.0.98.2:41283)
0929 16:26:33.922939 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:33.922944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:33.922963 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:26:33.923069 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:33.923072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:33.923128 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:33.923247 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:33.923512 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.924112 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.925094 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.925490 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.925594 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.925931 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:33.926485 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:33.926603 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:33.926711 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 16:26:33.926779 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:26:33.929943 8092 raft_consensus.cc:1261] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 1 FOLLOWER]: Refusing update from remote peer 2ba27a37ecd048cf95328fc90b9b6565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:33.930140 8372 raft_consensus.cc:1261] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a361147ca2454ab89c273e9bbd7fcdee [term 1 FOLLOWER]: Refusing update from remote peer 2ba27a37ecd048cf95328fc90b9b6565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:33.930534 8477 consensus_queue.cc:1035] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:33.930703 8454 consensus_queue.cc:1035] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:33.932411 8464 tablet_replica.cc:439] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a361147ca2454ab89c273e9bbd7fcdee: TxnStatusTablet state changed. Reason: New leader 2ba27a37ecd048cf95328fc90b9b6565. Latest consensus state: current_term: 1 leader_uuid: "2ba27a37ecd048cf95328fc90b9b6565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } } }
I20260929 16:26:33.932534 8464 tablet_replica.cc:442] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a361147ca2454ab89c273e9bbd7fcdee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:26:33.932489 8453 tablet_replica.cc:439] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: TxnStatusTablet state changed. Reason: New leader 2ba27a37ecd048cf95328fc90b9b6565. Latest consensus state: current_term: 1 leader_uuid: "2ba27a37ecd048cf95328fc90b9b6565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } } }
I20260929 16:26:33.932606 8453 tablet_replica.cc:442] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:26:33.934496 8454 tablet_replica.cc:439] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ba27a37ecd048cf95328fc90b9b6565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } } }
I20260929 16:26:33.934617 8454 tablet_replica.cc:442] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:26:33.934733 8454 tablet_replica.cc:439] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ba27a37ecd048cf95328fc90b9b6565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } } }
I20260929 16:26:33.934799 8454 tablet_replica.cc:442] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P 2ba27a37ecd048cf95328fc90b9b6565: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:26:33.935415 8453 tablet_replica.cc:439] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ba27a37ecd048cf95328fc90b9b6565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } } }
I20260929 16:26:33.935591 8453 tablet_replica.cc:442] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:26:33.936144 8464 tablet_replica.cc:439] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a361147ca2454ab89c273e9bbd7fcdee: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ba27a37ecd048cf95328fc90b9b6565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } } }
I20260929 16:26:33.936273 8464 tablet_replica.cc:442] T 6bad9d10be1a42f08f4ca81a8fcb1d87 P a361147ca2454ab89c273e9bbd7fcdee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:26:33.938408 8459 mvcc.cc:205] Tried to move back new op lower bound from 7334703898332053504 to 7334703898226905088. Current Snapshot: MvccSnapshot[applied={T|T < 7334703898332053504 or (T in {7334703898332053504})}]
I20260929 16:26:33.943691 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.940335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35328 (local address 127.0.98.2:41283)
0929 16:26:33.940452 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:33.940456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:33.940469 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:26:33.940571 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:33.940573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:33.940626 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:33.940688 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:33.940974 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.941545 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.942690 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.942895 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.942991 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.943251 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:33.943322 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:33.943371 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:33.943482 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 16:26:33.943525 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:26:33.954985 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.951560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46909 (local address 127.0.98.2:41283)
0929 16:26:33.951705 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:33.951710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:33.951886 (+ 176us) server_negotiation.cc:407] Connection header received
0929 16:26:33.952072 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:33.952076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:33.952120 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:33.952197 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:33.952480 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.952975 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.953911 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.954130 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.954235 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.954501 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:33.954576 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:33.954658 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:33.954738 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 16:26:33.954822 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":42}
I20260929 16:26:33.954991 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:33.951510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41283 (local address 127.0.98.2:46909)
0929 16:26:33.951803 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:33.951815 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:33.951973 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:33.952203 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:33.952206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:33.952212 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:26:33.952347 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.952352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.953087 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.953091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.953794 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:33.953804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:33.954239 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:33.954242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:33.954329 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:33.954377 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:26:33.954659 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:26:33.954665 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:26:33.954730 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 16:26:33.954849 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":25,"thread_start_us":113,"threads_started":1}
I20260929 16:26:33.962363 8372 raft_consensus.cc:1261] T 209f9dfe06534bfcafa9b1122c6bcf17 P a361147ca2454ab89c273e9bbd7fcdee [term 1 FOLLOWER]: Refusing update from remote peer 2ba27a37ecd048cf95328fc90b9b6565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:33.962363 8092 raft_consensus.cc:1261] T 209f9dfe06534bfcafa9b1122c6bcf17 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 1 FOLLOWER]: Refusing update from remote peer 2ba27a37ecd048cf95328fc90b9b6565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:33.962754 8477 consensus_queue.cc:1035] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:33.962949 8454 consensus_queue.cc:1035] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:33.966104 8460 mvcc.cc:205] Tried to move back new op lower bound from 7334703898466504704 to 7334703897659297792. Current Snapshot: MvccSnapshot[applied={T|T < 7334703898466504704}]
I20260929 16:26:34.001374 7934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:34.001557 7934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:34.003718 7934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b: Sending DeleteTablet for 3 replicas of tablet 209f9dfe06534bfcafa9b1122c6bcf17
I20260929 16:26:34.004426 8072 tablet_service.cc:1514] Processing DeleteTablet for tablet 209f9dfe06534bfcafa9b1122c6bcf17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39572
I20260929 16:26:34.004451 8352 tablet_service.cc:1514] Processing DeleteTablet for tablet 209f9dfe06534bfcafa9b1122c6bcf17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34910
I20260929 16:26:34.004618 8471 tablet_replica.cc:330] T 209f9dfe06534bfcafa9b1122c6bcf17 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: stopping tablet replica
I20260929 16:26:34.004662 8469 tablet_replica.cc:330] T 209f9dfe06534bfcafa9b1122c6bcf17 P a361147ca2454ab89c273e9bbd7fcdee: stopping tablet replica
I20260929 16:26:34.004649 8212 tablet_service.cc:1514] Processing DeleteTablet for tablet 209f9dfe06534bfcafa9b1122c6bcf17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35316
I20260929 16:26:34.004736 8471 raft_consensus.cc:2229] T 209f9dfe06534bfcafa9b1122c6bcf17 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:34.004781 8469 raft_consensus.cc:2229] T 209f9dfe06534bfcafa9b1122c6bcf17 P a361147ca2454ab89c273e9bbd7fcdee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:34.004813 8470 tablet_replica.cc:330] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565: stopping tablet replica
I20260929 16:26:34.004860 8471 raft_consensus.cc:2258] T 209f9dfe06534bfcafa9b1122c6bcf17 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:34.004895 8469 raft_consensus.cc:2258] T 209f9dfe06534bfcafa9b1122c6bcf17 P a361147ca2454ab89c273e9bbd7fcdee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:34.004921 8470 raft_consensus.cc:2229] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:26:34.005257 8470 raft_consensus.cc:2258] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:34.005820 8471 ts_tablet_manager.cc:1905] T 209f9dfe06534bfcafa9b1122c6bcf17 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:34.005956 8469 ts_tablet_manager.cc:1905] T 209f9dfe06534bfcafa9b1122c6bcf17 P a361147ca2454ab89c273e9bbd7fcdee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:34.008478 8469 ts_tablet_manager.cc:1918] T 209f9dfe06534bfcafa9b1122c6bcf17 P a361147ca2454ab89c273e9bbd7fcdee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:26:34.008574 8469 log.cc:1196] T 209f9dfe06534bfcafa9b1122c6bcf17 P a361147ca2454ab89c273e9bbd7fcdee: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/209f9dfe06534bfcafa9b1122c6bcf17
I20260929 16:26:34.008730 8471 ts_tablet_manager.cc:1918] T 209f9dfe06534bfcafa9b1122c6bcf17 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:26:34.008822 8471 log.cc:1196] T 209f9dfe06534bfcafa9b1122c6bcf17 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-0/wal/wals/209f9dfe06534bfcafa9b1122c6bcf17
I20260929 16:26:34.008858 8469 ts_tablet_manager.cc:1939] T 209f9dfe06534bfcafa9b1122c6bcf17 P a361147ca2454ab89c273e9bbd7fcdee: Deleting consensus metadata
I20260929 16:26:34.009081 8471 ts_tablet_manager.cc:1939] T 209f9dfe06534bfcafa9b1122c6bcf17 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Deleting consensus metadata
I20260929 16:26:34.009708 7920 catalog_manager.cc:4841] TS a361147ca2454ab89c273e9bbd7fcdee (127.0.98.3:37993): tablet 209f9dfe06534bfcafa9b1122c6bcf17 (table test-table [id=97db24d93a77492ab4cecc9a3c618d9a]) successfully deleted
I20260929 16:26:34.010035 8470 ts_tablet_manager.cc:1905] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:34.012241 8470 ts_tablet_manager.cc:1918] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:26:34.012310 8470 log.cc:1196] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/209f9dfe06534bfcafa9b1122c6bcf17
I20260929 16:26:34.012482 7921 catalog_manager.cc:4841] TS a3c12d4976bc4aa3a67d4ff1a5ab7a57 (127.0.98.1:46171): tablet 209f9dfe06534bfcafa9b1122c6bcf17 (table test-table [id=97db24d93a77492ab4cecc9a3c618d9a]) successfully deleted
I20260929 16:26:34.012560 8470 ts_tablet_manager.cc:1939] T 209f9dfe06534bfcafa9b1122c6bcf17 P 2ba27a37ecd048cf95328fc90b9b6565: Deleting consensus metadata
I20260929 16:26:34.013314 7920 catalog_manager.cc:4841] TS 2ba27a37ecd048cf95328fc90b9b6565 (127.0.98.2:41283): tablet 209f9dfe06534bfcafa9b1122c6bcf17 (table test-table [id=97db24d93a77492ab4cecc9a3c618d9a]) successfully deleted
Sep 29 16:26:34 dist-test-slave-vq9v krb5kdc[7890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:26:34 dist-test-slave-vq9v krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:34 dist-test-slave-vq9v krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:26:34.039497 7984 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:34.031967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53076 (local address 127.0.98.62:38557)
0929 16:26:34.032127 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:34.032130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:34.032245 (+ 115us) server_negotiation.cc:407] Connection header received
0929 16:26:34.032398 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:34.032400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:34.032446 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:34.032518 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:34.033431 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:34.033952 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:34.035232 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:34.035542 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:34.035648 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:34.039220 (+ 3572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53076: BlockingRecv error: recv got EOF from 127.0.0.1:53076 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 16:26:34 dist-test-slave-vq9v krb5kdc[7890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 16:26:34.051088 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:34.041573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53084 (local address 127.0.98.62:38557)
0929 16:26:34.041692 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:34.041697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:34.041731 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:26:34.041899 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:34.041911 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:34.041955 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:34.042016 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:34.042723 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:34.043230 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:34.044406 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:34.044648 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:34.044782 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:34.047666 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:34.047670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:34.047672 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:34.047701 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:34.049129 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:34.049628 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:34.049633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:34.049635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:34.049722 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:34.050013 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:34.050018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:34.050019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:34.050276 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:34.050438 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:34.050779 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 16:26:34.050918 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 16:26:34.055068 7934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:26:34.055576 7934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:26:34.060642 8072 tablet_service.cc:1467] Processing CreateTablet for tablet 6616b666844a4203afba69f3631b3cda (DEFAULT_TABLE table=test-table [id=ff0930ff3c404ccbb4ad8e654051c5df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:34.060797 8212 tablet_service.cc:1467] Processing CreateTablet for tablet 6616b666844a4203afba69f3631b3cda (DEFAULT_TABLE table=test-table [id=ff0930ff3c404ccbb4ad8e654051c5df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:34.061033 8072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6616b666844a4203afba69f3631b3cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:34.061064 8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6616b666844a4203afba69f3631b3cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:34.061683 8352 tablet_service.cc:1467] Processing CreateTablet for tablet 6616b666844a4203afba69f3631b3cda (DEFAULT_TABLE table=test-table [id=ff0930ff3c404ccbb4ad8e654051c5df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:34.061946 8352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6616b666844a4203afba69f3631b3cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:34.063963 8447 tablet_bootstrap.cc:492] T 6616b666844a4203afba69f3631b3cda P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Bootstrap starting.
I20260929 16:26:34.064373 8446 tablet_bootstrap.cc:492] T 6616b666844a4203afba69f3631b3cda P a361147ca2454ab89c273e9bbd7fcdee: Bootstrap starting.
I20260929 16:26:34.064477 8448 tablet_bootstrap.cc:492] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565: Bootstrap starting.
I20260929 16:26:34.064998 8447 tablet_bootstrap.cc:654] T 6616b666844a4203afba69f3631b3cda P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:34.065367 8448 tablet_bootstrap.cc:654] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:34.065431 8446 tablet_bootstrap.cc:654] T 6616b666844a4203afba69f3631b3cda P a361147ca2454ab89c273e9bbd7fcdee: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:34.067212 8447 tablet_bootstrap.cc:492] T 6616b666844a4203afba69f3631b3cda P a3c12d4976bc4aa3a67d4ff1a5ab7a57: No bootstrap required, opened a new log
I20260929 16:26:34.067315 8447 ts_tablet_manager.cc:1397] T 6616b666844a4203afba69f3631b3cda P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:26:34.068192 8447 raft_consensus.cc:348] T 6616b666844a4203afba69f3631b3cda P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:34.068334 8447 raft_consensus.cc:374] T 6616b666844a4203afba69f3631b3cda P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:34.068382 8447 raft_consensus.cc:729] T 6616b666844a4203afba69f3631b3cda P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c12d4976bc4aa3a67d4ff1a5ab7a57, State: Initialized, Role: FOLLOWER
I20260929 16:26:34.068514 8447 consensus_queue.cc:260] T 6616b666844a4203afba69f3631b3cda P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:34.068827 8447 ts_tablet_manager.cc:1428] T 6616b666844a4203afba69f3631b3cda P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 16:26:34.071069 8448 tablet_bootstrap.cc:492] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565: No bootstrap required, opened a new log
I20260929 16:26:34.071184 8448 ts_tablet_manager.cc:1397] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260929 16:26:34.071696 8448 raft_consensus.cc:348] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:34.071826 8448 raft_consensus.cc:374] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:34.071866 8448 raft_consensus.cc:729] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba27a37ecd048cf95328fc90b9b6565, State: Initialized, Role: FOLLOWER
I20260929 16:26:34.072029 8448 consensus_queue.cc:260] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:34.072352 8448 ts_tablet_manager.cc:1428] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 16:26:34.072444 8446 tablet_bootstrap.cc:492] T 6616b666844a4203afba69f3631b3cda P a361147ca2454ab89c273e9bbd7fcdee: No bootstrap required, opened a new log
I20260929 16:26:34.072541 8446 ts_tablet_manager.cc:1397] T 6616b666844a4203afba69f3631b3cda P a361147ca2454ab89c273e9bbd7fcdee: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260929 16:26:34.072996 8446 raft_consensus.cc:348] T 6616b666844a4203afba69f3631b3cda P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:34.073117 8446 raft_consensus.cc:374] T 6616b666844a4203afba69f3631b3cda P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:34.073153 8446 raft_consensus.cc:729] T 6616b666844a4203afba69f3631b3cda P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a361147ca2454ab89c273e9bbd7fcdee, State: Initialized, Role: FOLLOWER
I20260929 16:26:34.073318 8446 consensus_queue.cc:260] T 6616b666844a4203afba69f3631b3cda P a361147ca2454ab89c273e9bbd7fcdee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:34.074214 8446 ts_tablet_manager.cc:1428] T 6616b666844a4203afba69f3631b3cda P a361147ca2454ab89c273e9bbd7fcdee: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 16:26:34.156440 8500 raft_consensus.cc:482] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:26:34.156602 8500 raft_consensus.cc:504] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:34.157091 8500 leader_election.cc:288] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3c12d4976bc4aa3a67d4ff1a5ab7a57 (127.0.98.1:46171), a361147ca2454ab89c273e9bbd7fcdee (127.0.98.3:37993)
I20260929 16:26:34.157580 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6616b666844a4203afba69f3631b3cda" candidate_uuid: "2ba27a37ecd048cf95328fc90b9b6565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" is_pre_election: true
I20260929 16:26:34.157573 8372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6616b666844a4203afba69f3631b3cda" candidate_uuid: "2ba27a37ecd048cf95328fc90b9b6565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a361147ca2454ab89c273e9bbd7fcdee" is_pre_election: true
I20260929 16:26:34.157732 8372 raft_consensus.cc:2454] T 6616b666844a4203afba69f3631b3cda P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ba27a37ecd048cf95328fc90b9b6565 in term 0.
I20260929 16:26:34.157732 8091 raft_consensus.cc:2454] T 6616b666844a4203afba69f3631b3cda P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ba27a37ecd048cf95328fc90b9b6565 in term 0.
I20260929 16:26:34.158033 8164 leader_election.cc:302] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba27a37ecd048cf95328fc90b9b6565, a361147ca2454ab89c273e9bbd7fcdee; no voters:
I20260929 16:26:34.158211 8500 raft_consensus.cc:2781] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:26:34.158298 8500 raft_consensus.cc:482] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:26:34.158352 8500 raft_consensus.cc:3037] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:34.159355 8500 raft_consensus.cc:504] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:34.159724 8500 leader_election.cc:288] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565 [CANDIDATE]: Term 1 election: Requested vote from peers a3c12d4976bc4aa3a67d4ff1a5ab7a57 (127.0.98.1:46171), a361147ca2454ab89c273e9bbd7fcdee (127.0.98.3:37993)
I20260929 16:26:34.160053 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6616b666844a4203afba69f3631b3cda" candidate_uuid: "2ba27a37ecd048cf95328fc90b9b6565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57"
I20260929 16:26:34.160187 8091 raft_consensus.cc:3037] T 6616b666844a4203afba69f3631b3cda P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:34.160120 8372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6616b666844a4203afba69f3631b3cda" candidate_uuid: "2ba27a37ecd048cf95328fc90b9b6565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a361147ca2454ab89c273e9bbd7fcdee"
I20260929 16:26:34.160274 8372 raft_consensus.cc:3037] T 6616b666844a4203afba69f3631b3cda P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:34.161410 8372 raft_consensus.cc:2454] T 6616b666844a4203afba69f3631b3cda P a361147ca2454ab89c273e9bbd7fcdee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ba27a37ecd048cf95328fc90b9b6565 in term 1.
I20260929 16:26:34.161430 8091 raft_consensus.cc:2454] T 6616b666844a4203afba69f3631b3cda P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ba27a37ecd048cf95328fc90b9b6565 in term 1.
I20260929 16:26:34.161728 8165 leader_election.cc:302] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba27a37ecd048cf95328fc90b9b6565, a3c12d4976bc4aa3a67d4ff1a5ab7a57; no voters:
I20260929 16:26:34.161892 8500 raft_consensus.cc:2781] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:34.162034 8500 raft_consensus.cc:686] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565 [term 1 LEADER]: Becoming Leader. State: Replica: 2ba27a37ecd048cf95328fc90b9b6565, State: Running, Role: LEADER
I20260929 16:26:34.162185 8500 consensus_queue.cc:237] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:34.164062 7934 catalog_manager.cc:5495] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ba27a37ecd048cf95328fc90b9b6565 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2ba27a37ecd048cf95328fc90b9b6565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } health_report { overall_health: UNKNOWN } } }
I20260929 16:26:34.176680 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:34.172718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.0.98.2:41283)
0929 16:26:34.172842 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:34.172847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:34.172862 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:26:34.172956 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:34.172959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:34.173011 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:34.173085 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:34.173392 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:34.174168 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:34.175145 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:34.175483 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:34.175586 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:34.176003 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:34.176122 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:34.176229 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:34.176405 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 16:26:34.176490 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:26:34.179085 8091 raft_consensus.cc:1261] T 6616b666844a4203afba69f3631b3cda P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 1 FOLLOWER]: Refusing update from remote peer 2ba27a37ecd048cf95328fc90b9b6565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:34.179175 8372 raft_consensus.cc:1261] T 6616b666844a4203afba69f3631b3cda P a361147ca2454ab89c273e9bbd7fcdee [term 1 FOLLOWER]: Refusing update from remote peer 2ba27a37ecd048cf95328fc90b9b6565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:34.179677 8503 consensus_queue.cc:1035] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:34.179895 8500 consensus_queue.cc:1035] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:34.187820 7934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:34.187973 7934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:34.190395 7934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b: Sending DeleteTablet for 3 replicas of tablet 6616b666844a4203afba69f3631b3cda
I20260929 16:26:34.191054 8212 tablet_service.cc:1514] Processing DeleteTablet for tablet 6616b666844a4203afba69f3631b3cda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35316
I20260929 16:26:34.191077 8072 tablet_service.cc:1514] Processing DeleteTablet for tablet 6616b666844a4203afba69f3631b3cda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39572
I20260929 16:26:34.191224 8470 tablet_replica.cc:330] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565: stopping tablet replica
I20260929 16:26:34.191334 8470 raft_consensus.cc:2229] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:26:34.191516 8470 raft_consensus.cc:2258] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:34.191699 8471 tablet_replica.cc:330] T 6616b666844a4203afba69f3631b3cda P a3c12d4976bc4aa3a67d4ff1a5ab7a57: stopping tablet replica
I20260929 16:26:34.191811 8471 raft_consensus.cc:2229] T 6616b666844a4203afba69f3631b3cda P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:34.191959 8471 raft_consensus.cc:2258] T 6616b666844a4203afba69f3631b3cda P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:34.192286 7934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 16:26:34.192756 8471 ts_tablet_manager.cc:1905] T 6616b666844a4203afba69f3631b3cda P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 16:26:34.192858 7934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:26:34.195168 8471 ts_tablet_manager.cc:1918] T 6616b666844a4203afba69f3631b3cda P a3c12d4976bc4aa3a67d4ff1a5ab7a57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:26:34.195256 8471 log.cc:1196] T 6616b666844a4203afba69f3631b3cda P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-0/wal/wals/6616b666844a4203afba69f3631b3cda
I20260929 16:26:34.195503 8471 ts_tablet_manager.cc:1939] T 6616b666844a4203afba69f3631b3cda P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Deleting consensus metadata
I20260929 16:26:34.196033 8470 ts_tablet_manager.cc:1905] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:34.198207 8351 tablet_service.cc:1514] Processing DeleteTablet for tablet 6616b666844a4203afba69f3631b3cda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34910
I20260929 16:26:34.198380 8352 tablet_service.cc:1514] Processing DeleteTablet for tablet 6616b666844a4203afba69f3631b3cda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34910
I20260929 16:26:34.198458 8470 ts_tablet_manager.cc:1918] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:26:34.198482 8469 tablet_replica.cc:330] T 6616b666844a4203afba69f3631b3cda P a361147ca2454ab89c273e9bbd7fcdee: stopping tablet replica
I20260929 16:26:34.198523 8470 log.cc:1196] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/6616b666844a4203afba69f3631b3cda
I20260929 16:26:34.198632 8469 raft_consensus.cc:2229] T 6616b666844a4203afba69f3631b3cda P a361147ca2454ab89c273e9bbd7fcdee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:34.198770 8470 ts_tablet_manager.cc:1939] T 6616b666844a4203afba69f3631b3cda P 2ba27a37ecd048cf95328fc90b9b6565: Deleting consensus metadata
I20260929 16:26:34.198766 8469 raft_consensus.cc:2258] T 6616b666844a4203afba69f3631b3cda P a361147ca2454ab89c273e9bbd7fcdee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:34.199411 7920 catalog_manager.cc:4841] TS 2ba27a37ecd048cf95328fc90b9b6565 (127.0.98.2:41283): tablet 6616b666844a4203afba69f3631b3cda (table test-table [id=ff0930ff3c404ccbb4ad8e654051c5df]) successfully deleted
I20260929 16:26:34.200309 8469 ts_tablet_manager.cc:1905] T 6616b666844a4203afba69f3631b3cda P a361147ca2454ab89c273e9bbd7fcdee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:34.200590 7921 catalog_manager.cc:4841] TS a3c12d4976bc4aa3a67d4ff1a5ab7a57 (127.0.98.1:46171): tablet 6616b666844a4203afba69f3631b3cda (table test-table [id=ff0930ff3c404ccbb4ad8e654051c5df]) successfully deleted
I20260929 16:26:34.202766 8469 ts_tablet_manager.cc:1918] T 6616b666844a4203afba69f3631b3cda P a361147ca2454ab89c273e9bbd7fcdee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:26:34.202847 8469 log.cc:1196] T 6616b666844a4203afba69f3631b3cda P a361147ca2454ab89c273e9bbd7fcdee: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/6616b666844a4203afba69f3631b3cda
I20260929 16:26:34.202795 8072 tablet_service.cc:1467] Processing CreateTablet for tablet 673265bd162d40698725c463bb475563 (DEFAULT_TABLE table=test-table [id=1c57fed186a540459ba6262bfb1d7f88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:34.203083 8469 ts_tablet_manager.cc:1939] T 6616b666844a4203afba69f3631b3cda P a361147ca2454ab89c273e9bbd7fcdee: Deleting consensus metadata
I20260929 16:26:34.203096 8072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 673265bd162d40698725c463bb475563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:34.203142 8212 tablet_service.cc:1467] Processing CreateTablet for tablet 673265bd162d40698725c463bb475563 (DEFAULT_TABLE table=test-table [id=1c57fed186a540459ba6262bfb1d7f88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:34.203442 8212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 673265bd162d40698725c463bb475563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:34.206791 7920 catalog_manager.cc:4841] TS a361147ca2454ab89c273e9bbd7fcdee (127.0.98.3:37993): tablet 6616b666844a4203afba69f3631b3cda (table test-table [id=ff0930ff3c404ccbb4ad8e654051c5df]) successfully deleted
W20260929 16:26:34.207185 7920 catalog_manager.cc:4808] TS a361147ca2454ab89c273e9bbd7fcdee (127.0.98.3:37993): delete failed for tablet 6616b666844a4203afba69f3631b3cda because the tablet was not found. No further retry: Not found: Tablet not found: 6616b666844a4203afba69f3631b3cda
I20260929 16:26:34.207651 8352 tablet_service.cc:1467] Processing CreateTablet for tablet 673265bd162d40698725c463bb475563 (DEFAULT_TABLE table=test-table [id=1c57fed186a540459ba6262bfb1d7f88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:34.207991 8352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 673265bd162d40698725c463bb475563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:34.210410 8447 tablet_bootstrap.cc:492] T 673265bd162d40698725c463bb475563 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Bootstrap starting.
I20260929 16:26:34.211220 8446 tablet_bootstrap.cc:492] T 673265bd162d40698725c463bb475563 P a361147ca2454ab89c273e9bbd7fcdee: Bootstrap starting.
I20260929 16:26:34.211602 8447 tablet_bootstrap.cc:654] T 673265bd162d40698725c463bb475563 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:34.211602 8448 tablet_bootstrap.cc:492] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565: Bootstrap starting.
I20260929 16:26:34.212044 8446 tablet_bootstrap.cc:654] T 673265bd162d40698725c463bb475563 P a361147ca2454ab89c273e9bbd7fcdee: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:34.212715 8448 tablet_bootstrap.cc:654] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:34.213325 8446 tablet_bootstrap.cc:492] T 673265bd162d40698725c463bb475563 P a361147ca2454ab89c273e9bbd7fcdee: No bootstrap required, opened a new log
I20260929 16:26:34.213436 8446 ts_tablet_manager.cc:1397] T 673265bd162d40698725c463bb475563 P a361147ca2454ab89c273e9bbd7fcdee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 16:26:34.213929 8446 raft_consensus.cc:348] T 673265bd162d40698725c463bb475563 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:34.214061 8446 raft_consensus.cc:374] T 673265bd162d40698725c463bb475563 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:34.214089 8446 raft_consensus.cc:729] T 673265bd162d40698725c463bb475563 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a361147ca2454ab89c273e9bbd7fcdee, State: Initialized, Role: FOLLOWER
I20260929 16:26:34.214210 8446 consensus_queue.cc:260] T 673265bd162d40698725c463bb475563 P a361147ca2454ab89c273e9bbd7fcdee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:34.214481 8446 ts_tablet_manager.cc:1428] T 673265bd162d40698725c463bb475563 P a361147ca2454ab89c273e9bbd7fcdee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:26:34.214988 8447 tablet_bootstrap.cc:492] T 673265bd162d40698725c463bb475563 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: No bootstrap required, opened a new log
I20260929 16:26:34.215080 8447 ts_tablet_manager.cc:1397] T 673265bd162d40698725c463bb475563 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 16:26:34.215205 8448 tablet_bootstrap.cc:492] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565: No bootstrap required, opened a new log
I20260929 16:26:34.215296 8448 ts_tablet_manager.cc:1397] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 16:26:34.215570 8447 raft_consensus.cc:348] T 673265bd162d40698725c463bb475563 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:34.215692 8447 raft_consensus.cc:374] T 673265bd162d40698725c463bb475563 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:34.215758 8447 raft_consensus.cc:729] T 673265bd162d40698725c463bb475563 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c12d4976bc4aa3a67d4ff1a5ab7a57, State: Initialized, Role: FOLLOWER
I20260929 16:26:34.215708 8448 raft_consensus.cc:348] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:34.215821 8448 raft_consensus.cc:374] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:34.215859 8448 raft_consensus.cc:729] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba27a37ecd048cf95328fc90b9b6565, State: Initialized, Role: FOLLOWER
I20260929 16:26:34.215907 8447 consensus_queue.cc:260] T 673265bd162d40698725c463bb475563 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:34.215973 8448 consensus_queue.cc:260] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:34.216202 8447 ts_tablet_manager.cc:1428] T 673265bd162d40698725c463bb475563 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 16:26:34.216248 8448 ts_tablet_manager.cc:1428] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:26:34.375191 8503 raft_consensus.cc:482] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:26:34.375404 8503 raft_consensus.cc:504] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:34.375912 8503 leader_election.cc:288] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3c12d4976bc4aa3a67d4ff1a5ab7a57 (127.0.98.1:46171), a361147ca2454ab89c273e9bbd7fcdee (127.0.98.3:37993)
I20260929 16:26:34.376426 8372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "673265bd162d40698725c463bb475563" candidate_uuid: "2ba27a37ecd048cf95328fc90b9b6565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a361147ca2454ab89c273e9bbd7fcdee" is_pre_election: true
I20260929 16:26:34.376494 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "673265bd162d40698725c463bb475563" candidate_uuid: "2ba27a37ecd048cf95328fc90b9b6565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" is_pre_election: true
I20260929 16:26:34.376586 8372 raft_consensus.cc:2454] T 673265bd162d40698725c463bb475563 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ba27a37ecd048cf95328fc90b9b6565 in term 0.
I20260929 16:26:34.376689 8091 raft_consensus.cc:2454] T 673265bd162d40698725c463bb475563 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ba27a37ecd048cf95328fc90b9b6565 in term 0.
I20260929 16:26:34.376951 8164 leader_election.cc:302] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba27a37ecd048cf95328fc90b9b6565, a361147ca2454ab89c273e9bbd7fcdee; no voters:
I20260929 16:26:34.377123 8503 raft_consensus.cc:2781] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:26:34.377187 8503 raft_consensus.cc:482] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:26:34.377259 8503 raft_consensus.cc:3037] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:34.378291 8503 raft_consensus.cc:504] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:34.378644 8503 leader_election.cc:288] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565 [CANDIDATE]: Term 1 election: Requested vote from peers a3c12d4976bc4aa3a67d4ff1a5ab7a57 (127.0.98.1:46171), a361147ca2454ab89c273e9bbd7fcdee (127.0.98.3:37993)
I20260929 16:26:34.379011 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "673265bd162d40698725c463bb475563" candidate_uuid: "2ba27a37ecd048cf95328fc90b9b6565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57"
I20260929 16:26:34.379045 8372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "673265bd162d40698725c463bb475563" candidate_uuid: "2ba27a37ecd048cf95328fc90b9b6565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a361147ca2454ab89c273e9bbd7fcdee"
I20260929 16:26:34.379150 8372 raft_consensus.cc:3037] T 673265bd162d40698725c463bb475563 P a361147ca2454ab89c273e9bbd7fcdee [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:34.379148 8091 raft_consensus.cc:3037] T 673265bd162d40698725c463bb475563 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:34.380293 8372 raft_consensus.cc:2454] T 673265bd162d40698725c463bb475563 P a361147ca2454ab89c273e9bbd7fcdee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ba27a37ecd048cf95328fc90b9b6565 in term 1.
I20260929 16:26:34.380326 8091 raft_consensus.cc:2454] T 673265bd162d40698725c463bb475563 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ba27a37ecd048cf95328fc90b9b6565 in term 1.
I20260929 16:26:34.380621 8165 leader_election.cc:302] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ba27a37ecd048cf95328fc90b9b6565, a3c12d4976bc4aa3a67d4ff1a5ab7a57; no voters:
I20260929 16:26:34.380823 8503 raft_consensus.cc:2781] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:34.380911 8503 raft_consensus.cc:686] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565 [term 1 LEADER]: Becoming Leader. State: Replica: 2ba27a37ecd048cf95328fc90b9b6565, State: Running, Role: LEADER
I20260929 16:26:34.381047 8503 consensus_queue.cc:237] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } }
I20260929 16:26:34.382817 7934 catalog_manager.cc:5495] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ba27a37ecd048cf95328fc90b9b6565 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2ba27a37ecd048cf95328fc90b9b6565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ba27a37ecd048cf95328fc90b9b6565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 } health_report { overall_health: UNKNOWN } } }
I20260929 16:26:34.420620 8091 raft_consensus.cc:1261] T 673265bd162d40698725c463bb475563 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 1 FOLLOWER]: Refusing update from remote peer 2ba27a37ecd048cf95328fc90b9b6565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:34.420655 8372 raft_consensus.cc:1261] T 673265bd162d40698725c463bb475563 P a361147ca2454ab89c273e9bbd7fcdee [term 1 FOLLOWER]: Refusing update from remote peer 2ba27a37ecd048cf95328fc90b9b6565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:34.421022 8477 consensus_queue.cc:1035] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3c12d4976bc4aa3a67d4ff1a5ab7a57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:34.421223 8500 consensus_queue.cc:1035] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a361147ca2454ab89c273e9bbd7fcdee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:34.453775 7934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:34.453977 7934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:34.456032 7934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29fc8946cc8b4a299a1be3ef8cc4815b: Sending DeleteTablet for 3 replicas of tablet 673265bd162d40698725c463bb475563
I20260929 16:26:34.456619 8072 tablet_service.cc:1514] Processing DeleteTablet for tablet 673265bd162d40698725c463bb475563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39572
I20260929 16:26:34.456687 8212 tablet_service.cc:1514] Processing DeleteTablet for tablet 673265bd162d40698725c463bb475563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35316
I20260929 16:26:34.456838 8471 tablet_replica.cc:330] T 673265bd162d40698725c463bb475563 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: stopping tablet replica
I20260929 16:26:34.456904 8352 tablet_service.cc:1514] Processing DeleteTablet for tablet 673265bd162d40698725c463bb475563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:34 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34910
I20260929 16:26:34.456982 8471 raft_consensus.cc:2229] T 673265bd162d40698725c463bb475563 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:34.457046 8470 tablet_replica.cc:330] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565: stopping tablet replica
I20260929 16:26:34.457086 8469 tablet_replica.cc:330] T 673265bd162d40698725c463bb475563 P a361147ca2454ab89c273e9bbd7fcdee: stopping tablet replica
I20260929 16:26:34.457100 8471 raft_consensus.cc:2258] T 673265bd162d40698725c463bb475563 P a3c12d4976bc4aa3a67d4ff1a5ab7a57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:34.457158 8470 raft_consensus.cc:2229] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:26:34.457186 8469 raft_consensus.cc:2229] T 673265bd162d40698725c463bb475563 P a361147ca2454ab89c273e9bbd7fcdee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:34.457315 8469 raft_consensus.cc:2258] T 673265bd162d40698725c463bb475563 P a361147ca2454ab89c273e9bbd7fcdee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:34.457391 8470 raft_consensus.cc:2258] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:34.458045 8469 ts_tablet_manager.cc:1905] T 673265bd162d40698725c463bb475563 P a361147ca2454ab89c273e9bbd7fcdee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:34.458137 8470 ts_tablet_manager.cc:1905] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:34.458164 8471 ts_tablet_manager.cc:1905] T 673265bd162d40698725c463bb475563 P a3c12d4976bc4aa3a67d4ff1a5ab7a57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:34.458844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 8007
I20260929 16:26:34.460536 8469 ts_tablet_manager.cc:1918] T 673265bd162d40698725c463bb475563 P a361147ca2454ab89c273e9bbd7fcdee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:26:34.460536 8470 ts_tablet_manager.cc:1918] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:26:34.460621 8470 log.cc:1196] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/673265bd162d40698725c463bb475563
I20260929 16:26:34.460620 8469 log.cc:1196] T 673265bd162d40698725c463bb475563 P a361147ca2454ab89c273e9bbd7fcdee: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/673265bd162d40698725c463bb475563
I20260929 16:26:34.460889 8469 ts_tablet_manager.cc:1939] T 673265bd162d40698725c463bb475563 P a361147ca2454ab89c273e9bbd7fcdee: Deleting consensus metadata
I20260929 16:26:34.460889 8470 ts_tablet_manager.cc:1939] T 673265bd162d40698725c463bb475563 P 2ba27a37ecd048cf95328fc90b9b6565: Deleting consensus metadata
I20260929 16:26:34.461989 7920 catalog_manager.cc:4841] TS a361147ca2454ab89c273e9bbd7fcdee (127.0.98.3:37993): tablet 673265bd162d40698725c463bb475563 (table test-table [id=1c57fed186a540459ba6262bfb1d7f88]) successfully deleted
I20260929 16:26:34.462176 7920 catalog_manager.cc:4841] TS 2ba27a37ecd048cf95328fc90b9b6565 (127.0.98.2:41283): tablet 673265bd162d40698725c463bb475563 (table test-table [id=1c57fed186a540459ba6262bfb1d7f88]) successfully deleted
W20260929 16:26:34.465734 7921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46171 (error 108)
I20260929 16:26:34.466758 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:34.466391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46171 (local address 127.0.0.1:39586)
0929 16:26:34.466568 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:34.466628 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":47,"threads_started":1}
I20260929 16:26:34.466781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 8147
W20260929 16:26:34.467597 7921 catalog_manager.cc:4584] TS a3c12d4976bc4aa3a67d4ff1a5ab7a57 (127.0.98.1:46171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 673265bd162d40698725c463bb475563: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46171: connect: Connection refused (error 111)
I20260929 16:26:34.474049 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 8287
I20260929 16:26:34.480304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 7903
2026-09-29T16:26:34Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3320 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:26:34 dist-test-slave-vq9v krb5kdc[8533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:26:34 dist-test-slave-vq9v krb5kdc[8533](info): set up 2 sockets
Sep 29 16:26:34 dist-test-slave-vq9v krb5kdc[8533](info): commencing operation
krb5kdc: starting...
W20260929 16:26:35.514016 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:26:35 dist-test-slave-vq9v krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:26:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:26:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:26:35.636421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39295
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:35.750720 8546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:35.751080 8546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:35.751154 8546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:35.755417 8546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:26:35.755535 8546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:35.755575 8546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:35.755611 8546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:26:35.755649 8546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:26:35.761477 8546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:35.762656 8546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:35.763728 8546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:35.772553 8551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:35.772559 8554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:35.772569 8552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:35.772877 8546 server_base.cc:1034] running on GCE node
I20260929 16:26:35.773597 8546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:35.774746 8546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:35.775933 8546 hybrid_clock.cc:648] HybridClock initialized: now 1790699195775915 us; error 37 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T16:26:35Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1304 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:26:35 dist-test-slave-vq9v krb5kdc[8559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:26:35 dist-test-slave-vq9v krb5kdc[8559](info): set up 2 sockets
Sep 29 16:26:35 dist-test-slave-vq9v krb5kdc[8559](info): commencing operation
krb5kdc: starting...
W20260929 16:26:36.850416 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:26:36 dist-test-slave-vq9v krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:26:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:26:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:26:36.981357 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46605
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:37.097574 8572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:37.097942 8572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:37.098060 8572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:37.103861 8572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:26:37.103953 8572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:37.103991 8572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:37.104027 8572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:26:37.104065 8572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:26:37.111867 8572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:37.113430 8572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:37.114678 8572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:37.125939 8578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:37.125949 8580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:37.125939 8577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:37.126433 8572 server_base.cc:1034] running on GCE node
I20260929 16:26:37.127224 8572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:37.128852 8572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:37.130033 8572 hybrid_clock.cc:648] HybridClock initialized: now 1790699197129921 us; error 136 us; skew 500 ppm
Sep 29 16:26:37 dist-test-slave-vq9v krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:37.133158 8572 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:26:37.134347 8572 webserver.cc:460] Webserver started at http://127.0.98.62:40119/ using document root <none> and password file <none>
I20260929 16:26:37.135092 8572 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:37.135178 8572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:37.135444 8572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:37.137756 8572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "830e8be0a655483883c37a35da35a6b3"
format_stamp: "Formatted at 2026-09-29 16:26:37 on dist-test-slave-vq9v"
server_key: "03efecd79694862b14ad19c0d0c33af7"
server_key_iv: "e56d6604137033686ea387d1e60c1f6a"
server_key_version: "encryptionkey@0"
I20260929 16:26:37.138381 8572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "830e8be0a655483883c37a35da35a6b3"
format_stamp: "Formatted at 2026-09-29 16:26:37 on dist-test-slave-vq9v"
server_key: "03efecd79694862b14ad19c0d0c33af7"
server_key_iv: "e56d6604137033686ea387d1e60c1f6a"
server_key_version: "encryptionkey@0"
I20260929 16:26:37.142765 8572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:26:37.145560 8587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:37.146943 8572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:37.147112 8572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "830e8be0a655483883c37a35da35a6b3"
format_stamp: "Formatted at 2026-09-29 16:26:37 on dist-test-slave-vq9v"
server_key: "03efecd79694862b14ad19c0d0c33af7"
server_key_iv: "e56d6604137033686ea387d1e60c1f6a"
server_key_version: "encryptionkey@0"
I20260929 16:26:37.147244 8572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:37.163362 8572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:37.166760 8572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:37.166976 8572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:37.174134 8572 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46605
I20260929 16:26:37.174157 8639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46605 every 8 connection(s)
I20260929 16:26:37.175163 8572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:26:37.178013 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 8572
I20260929 16:26:37.178082 8640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:37.178123 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:26:37.178465 392 external_mini_cluster.cc:1428] Setting key 29c5c6fdbcbeac013e8733eafae910dd
I20260929 16:26:37.185417 8640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3: Bootstrap starting.
Sep 29 16:26:37 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:37.188354 8640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:37.189116 8640 log.cc:824] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:37.191052 8640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3: No bootstrap required, opened a new log
I20260929 16:26:37.193831 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:37.179922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42526 (local address 127.0.98.62:46605)
0929 16:26:37.180356 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:37.180375 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:37.180411 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:26:37.181107 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:37.181124 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:37.181545 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:37.181838 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:37.182788 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:37.183837 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:37.185026 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:37.185628 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:37.185752 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:37.188710 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:37.188715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:37.188729 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:37.188758 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:37.190722 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:37.191259 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:37.191264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:37.191268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:37.191385 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:37.191713 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:37.191717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:37.191719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:37.192150 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:37.192366 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:37.192712 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 16:26:37.193017 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":153,"threads_started":1}
I20260929 16:26:37.194823 8640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830e8be0a655483883c37a35da35a6b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46605 } }
I20260929 16:26:37.195096 8640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:37.195154 8640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830e8be0a655483883c37a35da35a6b3, State: Initialized, Role: FOLLOWER
I20260929 16:26:37.195612 8640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830e8be0a655483883c37a35da35a6b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46605 } }
I20260929 16:26:37.195760 8640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:26:37.195822 8640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:26:37.195923 8640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:37.196851 8640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830e8be0a655483883c37a35da35a6b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46605 } }
I20260929 16:26:37.197177 8640 leader_election.cc:302] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 830e8be0a655483883c37a35da35a6b3; no voters:
I20260929 16:26:37.197584 8640 leader_election.cc:288] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:26:37.197727 8645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:37.197962 8645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [term 1 LEADER]: Becoming Leader. State: Replica: 830e8be0a655483883c37a35da35a6b3, State: Running, Role: LEADER
I20260929 16:26:37.198266 8645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830e8be0a655483883c37a35da35a6b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46605 } }
I20260929 16:26:37.198760 8640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:26:37.199934 8647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 830e8be0a655483883c37a35da35a6b3. Latest consensus state: current_term: 1 leader_uuid: "830e8be0a655483883c37a35da35a6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830e8be0a655483883c37a35da35a6b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46605 } } }
I20260929 16:26:37.200062 8647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:37.200510 8654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:26:37.204062 8654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:26:37.200361 8646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "830e8be0a655483883c37a35da35a6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830e8be0a655483883c37a35da35a6b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46605 } } }
I20260929 16:26:37.205420 8646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:37.209484 8654 catalog_manager.cc:1347] Generated new cluster ID: a8b477597df944babd900d5f05721dd3
I20260929 16:26:37.209571 8654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:26:37.226858 8654 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:26:37.227875 8654 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:26:37.236596 8654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3: Generated new TSK 0
I20260929 16:26:37.237386 8654 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 16:26:37.269274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46605
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:37.389078 8666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:37.389456 8666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:37.389529 8666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:37.393808 8666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:37.393908 8666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:37.394002 8666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:26:37.399660 8666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:37.400890 8666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:37.402026 8666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:37.412460 8671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:37.412496 8672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:37.412446 8674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:37.412925 8666 server_base.cc:1034] running on GCE node
I20260929 16:26:37.413468 8666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:37.414237 8666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:37.415436 8666 hybrid_clock.cc:648] HybridClock initialized: now 1790699197415418 us; error 38 us; skew 500 ppm
Sep 29 16:26:37 dist-test-slave-vq9v krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:37.418821 8666 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:26:37.419878 8666 webserver.cc:460] Webserver started at http://127.0.98.1:40337/ using document root <none> and password file <none>
I20260929 16:26:37.420446 8666 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:37.420509 8666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:37.420680 8666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:37.422905 8666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "08b7c4ab69ea48ada7ff60cba5854b98"
format_stamp: "Formatted at 2026-09-29 16:26:37 on dist-test-slave-vq9v"
server_key: "860d644b89964749246c838132c242f2"
server_key_iv: "a1119ed5fc0ad9ac58cc95d889dc38d9"
server_key_version: "encryptionkey@0"
I20260929 16:26:37.423470 8666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "08b7c4ab69ea48ada7ff60cba5854b98"
format_stamp: "Formatted at 2026-09-29 16:26:37 on dist-test-slave-vq9v"
server_key: "860d644b89964749246c838132c242f2"
server_key_iv: "a1119ed5fc0ad9ac58cc95d889dc38d9"
server_key_version: "encryptionkey@0"
I20260929 16:26:37.427764 8666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 16:26:37.430532 8681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:37.432053 8666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:26:37.432214 8666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "08b7c4ab69ea48ada7ff60cba5854b98"
format_stamp: "Formatted at 2026-09-29 16:26:37 on dist-test-slave-vq9v"
server_key: "860d644b89964749246c838132c242f2"
server_key_iv: "a1119ed5fc0ad9ac58cc95d889dc38d9"
server_key_version: "encryptionkey@0"
I20260929 16:26:37.432446 8666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:37.450879 8666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:37.454945 8666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:37.455173 8666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:37.455812 8666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:37.456703 8666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:37.456753 8666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:37.456820 8666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:37.456852 8666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:37.465643 8666 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39885
I20260929 16:26:37.465727 8794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39885 every 8 connection(s)
I20260929 16:26:37.466607 8666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:26:37.476610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 8666
Sep 29 16:26:37 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:37.476779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:26:37.477069 392 external_mini_cluster.cc:1428] Setting key ac274e61a3bc6d630e46a9ab18e868d8
I20260929 16:26:37.482086 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:37.468544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43583 (local address 127.0.98.62:46605)
0929 16:26:37.468677 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:37.468680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:37.469588 (+ 908us) server_negotiation.cc:407] Connection header received
0929 16:26:37.471053 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:37.471056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:37.471115 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:37.471227 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:37.472433 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:37.472972 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:37.474232 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:37.474467 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:37.474578 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:37.477527 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:37.477531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:37.477534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:37.477559 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:37.479277 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:37.479995 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:37.480021 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:37.480024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:37.480128 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:37.480658 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:37.480662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:37.480663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:37.480952 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:37.481060 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:37.481763 (+ 703us) server_negotiation.cc:299] Negotiation successful
0929 16:26:37.481924 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:26:37.483065 8797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:37.468906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.1:43583)
0929 16:26:37.469436 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:37.469480 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:37.470836 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:37.471401 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:37.471407 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:37.471775 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:37.472286 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:37.472300 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:37.473098 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:37.473107 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:37.474101 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:37.474110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:37.474618 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:37.474622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:37.474809 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:37.475481 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:37.475497 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:37.477314 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:37.479415 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:37.479426 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:37.479444 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:37.479855 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:37.480253 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:37.480258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:37.480260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:37.480483 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:37.481084 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:37.481089 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:37.481489 (+ 400us) client_negotiation.cc:769] Sending connection context
0929 16:26:37.481667 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 16:26:37.481957 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":117,"threads_started":1}
I20260929 16:26:37.484391 8795 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46605
I20260929 16:26:37.484661 8795 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:37.485360 8795 heartbeater.cc:511] Master 127.0.98.62:46605 requested a full tablet report, sending...
I20260929 16:26:37.487277 8604 ts_manager.cc:194] Registered new tserver with Master: 08b7c4ab69ea48ada7ff60cba5854b98 (127.0.98.1:39885)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 16:26:37.488909 8604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 16:26:37.509522 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46605
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:37.621121 8800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:37.621434 8800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:37.621527 8800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:37.625810 8800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:37.625885 8800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:37.625948 8800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:26:37.631592 8800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:37.632741 8800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:37.633941 8800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:37.643916 8806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:37.643922 8805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:37.644012 8800 server_base.cc:1034] running on GCE node
W20260929 16:26:37.643922 8808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:37.644793 8800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:37.645730 8800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:37.646924 8800 hybrid_clock.cc:648] HybridClock initialized: now 1790699197646893 us; error 46 us; skew 500 ppm
Sep 29 16:26:37 dist-test-slave-vq9v krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:37.650084 8800 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:26:37.651142 8800 webserver.cc:460] Webserver started at http://127.0.98.2:36411/ using document root <none> and password file <none>
I20260929 16:26:37.651731 8800 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:37.651809 8800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:37.652004 8800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:37.654233 8800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "c9f0e16b306c4ce8ab2441cfa4e48194"
format_stamp: "Formatted at 2026-09-29 16:26:37 on dist-test-slave-vq9v"
server_key: "0041c8f8e709df5b106468ea4c78197a"
server_key_iv: "ec847cc151e366df16d51f17d707bc5c"
server_key_version: "encryptionkey@0"
I20260929 16:26:37.654783 8800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c9f0e16b306c4ce8ab2441cfa4e48194"
format_stamp: "Formatted at 2026-09-29 16:26:37 on dist-test-slave-vq9v"
server_key: "0041c8f8e709df5b106468ea4c78197a"
server_key_iv: "ec847cc151e366df16d51f17d707bc5c"
server_key_version: "encryptionkey@0"
I20260929 16:26:37.658671 8800 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:26:37.661161 8815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:37.662485 8800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:26:37.662612 8800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "c9f0e16b306c4ce8ab2441cfa4e48194"
format_stamp: "Formatted at 2026-09-29 16:26:37 on dist-test-slave-vq9v"
server_key: "0041c8f8e709df5b106468ea4c78197a"
server_key_iv: "ec847cc151e366df16d51f17d707bc5c"
server_key_version: "encryptionkey@0"
I20260929 16:26:37.662720 8800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:37.673686 8800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:37.676615 8800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:37.676812 8800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:37.677459 8800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:37.678272 8800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:37.678318 8800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:37.678390 8800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:37.678426 8800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:37.687428 8800 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37981
I20260929 16:26:37.687450 8928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37981 every 8 connection(s)
I20260929 16:26:37.688326 8800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:26:37.696257 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 8800
I20260929 16:26:37.696403 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:26:37.696666 392 external_mini_cluster.cc:1428] Setting key 2a6be2d2cd23f5713a4e42c066523350
Sep 29 16:26:37 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:37.704524 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:37.690319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48085 (local address 127.0.98.62:46605)
0929 16:26:37.690503 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:37.690508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:37.691522 (+ 1014us) server_negotiation.cc:407] Connection header received
0929 16:26:37.693104 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:37.693108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:37.693175 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:37.693348 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:37.694657 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:37.695266 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:37.696593 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:37.696918 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:37.697050 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:37.700268 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:37.700275 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:37.700281 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:37.700316 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:37.702217 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:37.702800 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:37.702807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:37.702811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:37.702913 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:37.703249 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:37.703254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:37.703257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:37.703528 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:37.703649 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:37.704134 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 16:26:37.704282 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 16:26:37.705324 8931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:37.690796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.2:48085)
0929 16:26:37.691354 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:37.691404 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:37.692851 (+ 1447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:37.693535 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:37.693542 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:37.693951 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:37.694487 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:37.694502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:37.695430 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:37.695433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:37.696452 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:37.696464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:37.697092 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:37.697097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:37.697351 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:37.698075 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:37.698095 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:37.700085 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:37.702336 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:37.702341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:37.702356 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:37.702672 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:37.703020 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:37.703022 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:37.703023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:37.703160 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:37.703633 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:37.703637 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:37.703892 (+ 255us) client_negotiation.cc:769] Sending connection context
0929 16:26:37.704049 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 16:26:37.704318 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":109,"threads_started":1}
I20260929 16:26:37.706544 8929 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46605
I20260929 16:26:37.706827 8929 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 16:26:37.707449 8929 heartbeater.cc:511] Master 127.0.98.62:46605 requested a full tablet report, sending...
I20260929 16:26:37.708524 8604 ts_manager.cc:194] Registered new tserver with Master: c9f0e16b306c4ce8ab2441cfa4e48194 (127.0.98.2:37981)
I20260929 16:26:37.709113 8604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 16:26:37.728907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46605
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:37.846593 8934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:37.846868 8934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:37.846913 8934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:37.851300 8934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:37.851413 8934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:37.851490 8934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:26:37.857097 8934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:37.858250 8934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:37.859236 8934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:37.868922 8939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:37.868932 8940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:37.868937 8942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:37.869324 8934 server_base.cc:1034] running on GCE node
I20260929 16:26:37.869760 8934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:37.870405 8934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:37.871623 8934 hybrid_clock.cc:648] HybridClock initialized: now 1790699197871603 us; error 34 us; skew 500 ppm
Sep 29 16:26:37 dist-test-slave-vq9v krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:37.874706 8934 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:26:37.875711 8934 webserver.cc:460] Webserver started at http://127.0.98.3:34733/ using document root <none> and password file <none>
I20260929 16:26:37.876340 8934 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:37.876438 8934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:37.876677 8934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:37.878969 8934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "84ae6e1ccf6146e8a39dbd0ad536b272"
format_stamp: "Formatted at 2026-09-29 16:26:37 on dist-test-slave-vq9v"
server_key: "077187cc603e27b019b2d9531b990cdd"
server_key_iv: "06a48189e80d7b971214e39c9cdea9d6"
server_key_version: "encryptionkey@0"
I20260929 16:26:37.879590 8934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "84ae6e1ccf6146e8a39dbd0ad536b272"
format_stamp: "Formatted at 2026-09-29 16:26:37 on dist-test-slave-vq9v"
server_key: "077187cc603e27b019b2d9531b990cdd"
server_key_iv: "06a48189e80d7b971214e39c9cdea9d6"
server_key_version: "encryptionkey@0"
I20260929 16:26:37.883558 8934 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:26:37.886931 8949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:37.888335 8934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:26:37.888484 8934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "84ae6e1ccf6146e8a39dbd0ad536b272"
format_stamp: "Formatted at 2026-09-29 16:26:37 on dist-test-slave-vq9v"
server_key: "077187cc603e27b019b2d9531b990cdd"
server_key_iv: "06a48189e80d7b971214e39c9cdea9d6"
server_key_version: "encryptionkey@0"
I20260929 16:26:37.888622 8934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:37.908617 8934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:37.912057 8934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:37.912283 8934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:37.912917 8934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:37.913913 8934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:37.913986 8934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:37.914057 8934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:37.914106 8934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:37.924248 8934 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46523
I20260929 16:26:37.924270 9062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46523 every 8 connection(s)
I20260929 16:26:37.925279 8934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:26:37.926156 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 8934
I20260929 16:26:37.926307 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:26:37.926653 392 external_mini_cluster.cc:1428] Setting key 2d5bade64a140d9a3398f37931b326f7
Sep 29 16:26:37 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:37.939709 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:37.927264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44975 (local address 127.0.98.62:46605)
0929 16:26:37.927419 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:37.927424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:37.928499 (+ 1075us) server_negotiation.cc:407] Connection header received
0929 16:26:37.930023 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:37.930026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:37.930071 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:37.930141 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:37.931264 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:37.931798 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:37.932997 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:37.933265 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:37.933366 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:37.935940 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:37.935944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:37.935946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:37.935971 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:37.937421 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:37.938040 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:37.938043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:37.938045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:37.938146 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:37.938557 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:37.938560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:37.938562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:37.938772 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:37.938869 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:37.939399 (+ 530us) server_negotiation.cc:299] Negotiation successful
0929 16:26:37.939539 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:26:37.940724 9065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:37.927765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.3:44975)
0929 16:26:37.928352 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:37.928399 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:37.929790 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:37.930281 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:37.930288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:37.930644 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:37.931130 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:37.931144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:37.931912 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:37.931916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:37.932876 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:37.932885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:37.933404 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:37.933408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:37.933564 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:37.934184 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:37.934200 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:37.935759 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:37.937539 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:37.937545 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:37.937559 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:37.937910 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:37.938280 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:37.938282 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:37.938283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:37.938420 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:37.938872 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:37.938877 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:37.939146 (+ 269us) client_negotiation.cc:769] Sending connection context
0929 16:26:37.939351 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 16:26:37.939754 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":160,"threads_started":1}
I20260929 16:26:37.941831 9063 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46605
I20260929 16:26:37.942121 9063 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:37.942638 9063 heartbeater.cc:511] Master 127.0.98.62:46605 requested a full tablet report, sending...
I20260929 16:26:37.943691 8604 ts_manager.cc:194] Registered new tserver with Master: 84ae6e1ccf6146e8a39dbd0ad536b272 (127.0.98.3:46523)
I20260929 16:26:37.944316 8604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44975
I20260929 16:26:37.953150 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 16:26:38.030054 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 8666
I20260929 16:26:38.035987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 8800
I20260929 16:26:38.041668 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 8934
I20260929 16:26:38.046995 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 8572
I20260929 16:26:38.053574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46605
--webserver_interface=127.0.98.62
--webserver_port=40119
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:38.167385 9072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:38.167743 9072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:38.167820 9072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:38.172081 9072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:26:38.172194 9072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:38.172233 9072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:38.172267 9072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:26:38.172314 9072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:26:38.177944 9072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:38.179101 9072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:38.180150 9072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:38.189277 9078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:38.189301 9080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:38.189281 9072 server_base.cc:1034] running on GCE node
W20260929 16:26:38.189301 9077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:38.190222 9072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:38.191433 9072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:38.192629 9072 hybrid_clock.cc:648] HybridClock initialized: now 1790699198192608 us; error 36 us; skew 500 ppm
Sep 29 16:26:38 dist-test-slave-vq9v krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:38.195617 9072 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:26:38.196520 9072 webserver.cc:460] Webserver started at http://127.0.98.62:40119/ using document root <none> and password file <none>
I20260929 16:26:38.197046 9072 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:38.197119 9072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:38.201094 9072 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:26:38.203538 9087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:38.205021 9072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:26:38.205173 9072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "830e8be0a655483883c37a35da35a6b3"
format_stamp: "Formatted at 2026-09-29 16:26:37 on dist-test-slave-vq9v"
server_key: "03efecd79694862b14ad19c0d0c33af7"
server_key_iv: "e56d6604137033686ea387d1e60c1f6a"
server_key_version: "encryptionkey@0"
I20260929 16:26:38.206044 9072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:38.223858 9072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:38.227391 9072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:38.227594 9072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:38.234521 9072 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46605
I20260929 16:26:38.234579 9139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46605 every 8 connection(s)
I20260929 16:26:38.235651 9072 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:26:38.240146 9140 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:26:38.240486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 9072
I20260929 16:26:38.240993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39885
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46605
--webserver_port=40337
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:26:38.242053 9140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3: Bootstrap starting.
I20260929 16:26:38.265811 9140 log.cc:824] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:38.274447 9140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:26:38.274928 9140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3: Bootstrap complete.
I20260929 16:26:38.280519 9140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830e8be0a655483883c37a35da35a6b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46605 } }
I20260929 16:26:38.281039 9140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830e8be0a655483883c37a35da35a6b3, State: Initialized, Role: FOLLOWER
I20260929 16:26:38.281637 9140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830e8be0a655483883c37a35da35a6b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46605 } }
I20260929 16:26:38.281790 9140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:26:38.281860 9140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:26:38.281951 9140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:26:38.284642 9140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830e8be0a655483883c37a35da35a6b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46605 } }
I20260929 16:26:38.285033 9140 leader_election.cc:302] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 830e8be0a655483883c37a35da35a6b3; no voters:
I20260929 16:26:38.285471 9140 leader_election.cc:288] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:26:38.285610 9144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:26:38.285894 9144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [term 2 LEADER]: Becoming Leader. State: Replica: 830e8be0a655483883c37a35da35a6b3, State: Running, Role: LEADER
I20260929 16:26:38.286207 9144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830e8be0a655483883c37a35da35a6b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46605 } }
I20260929 16:26:38.286742 9140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:26:38.287187 9146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 830e8be0a655483883c37a35da35a6b3. Latest consensus state: current_term: 2 leader_uuid: "830e8be0a655483883c37a35da35a6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830e8be0a655483883c37a35da35a6b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46605 } } }
I20260929 16:26:38.287324 9146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:38.287647 9145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "830e8be0a655483883c37a35da35a6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "830e8be0a655483883c37a35da35a6b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46605 } } }
I20260929 16:26:38.287735 9145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:38.287856 9148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:26:38.291242 9148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:26:38.292013 9148 catalog_manager.cc:1259] Loaded cluster ID: a8b477597df944babd900d5f05721dd3
I20260929 16:26:38.292073 9148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:26:38.294404 9148 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:26:38.295121 9148 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 830e8be0a655483883c37a35da35a6b3: Loaded TSK: 0
I20260929 16:26:38.296382 9148 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:26:38.372180 9142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:38.372452 9142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:38.372498 9142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:38.376681 9142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:38.376761 9142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:38.376859 9142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:26:38.382269 9142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40337
--tserver_master_addrs=127.0.98.62:46605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:38.383340 9142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:38.384351 9142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:38.394511 9167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:38.394516 9166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:38.394608 9142 server_base.cc:1034] running on GCE node
W20260929 16:26:38.394515 9169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:38.395195 9142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:38.395980 9142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:38.397192 9142 hybrid_clock.cc:648] HybridClock initialized: now 1790699198397159 us; error 50 us; skew 500 ppm
Sep 29 16:26:38 dist-test-slave-vq9v krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699198, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:38.400111 9142 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 16:26:38.401191 9142 webserver.cc:460] Webserver started at http://127.0.98.1:40337/ using document root <none> and password file <none>
I20260929 16:26:38.401853 9142 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:38.401932 9142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:38.406069 9142 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 16:26:38.408438 9176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:38.409929 9142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:26:38.410065 9142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "08b7c4ab69ea48ada7ff60cba5854b98"
format_stamp: "Formatted at 2026-09-29 16:26:37 on dist-test-slave-vq9v"
server_key: "860d644b89964749246c838132c242f2"
server_key_iv: "a1119ed5fc0ad9ac58cc95d889dc38d9"
server_key_version: "encryptionkey@0"
I20260929 16:26:38.410691 9142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:38.437546 9142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:38.441409 9142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:38.441612 9142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:38.442222 9142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:38.443068 9142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:38.443114 9142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:38.443179 9142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:38.443210 9142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:38.452364 9142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39885
I20260929 16:26:38.452382 9289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39885 every 8 connection(s)
I20260929 16:26:38.453509 9142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:26:38.458770 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 9142
I20260929 16:26:38.459348 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37981
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46605
--webserver_port=36411
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:26:38 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:38 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:26:38.469573 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:38.455767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35381 (local address 127.0.98.62:46605)
0929 16:26:38.456245 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:38.456286 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:38.456573 (+ 287us) server_negotiation.cc:407] Connection header received
0929 16:26:38.458257 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:38.458269 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:38.458592 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:38.458921 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:38.460549 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.462251 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.464125 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.464600 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.464723 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:38.468734 (+ 4011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35381: BlockingRecv error: recv got EOF from 127.0.98.1:35381 (error 108)
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":116,"threads_started":1}
I20260929 16:26:38.469578 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:38.455931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.1:35381)
0929 16:26:38.456418 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:38.456461 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:38.457843 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:38.459043 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:38.459052 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:38.459621 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:38.460330 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:38.460352 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.462388 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.462395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:38.463987 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:38.464001 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.464772 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.464780 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:38.465018 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:38.465752 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:38.465781 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:38.468514 (+ 2733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":146,"threads_started":1}
W20260929 16:26:38.470291 9290 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46605 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:46605: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 16:26:38 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:38 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:38.476608 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:38.470789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.1:55729)
0929 16:26:38.470984 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:38.470999 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:38.471088 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:38.471358 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:38.471361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:38.471542 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:38.471644 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:38.471649 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.472415 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.472420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:38.473006 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:38.473014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.473570 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.473576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:38.473676 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:38.474208 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:38.474220 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:38.476453 (+ 2233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109}
W20260929 16:26:38.477054 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:38.470896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55729 (local address 127.0.98.62:46605)
0929 16:26:38.471030 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:38.471037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:38.471052 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:26:38.471189 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:38.471192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:38.471242 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:38.471334 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:38.471760 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.472299 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.473130 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.473437 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.473546 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:38.476661 (+ 3115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55729: BlockingRecv error: recv got EOF from 127.0.98.1:55729 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 16:26:38 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:38 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:38.483801 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:38.477238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.1:41881)
0929 16:26:38.477431 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:38.477444 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:38.477517 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:38.477765 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:38.477767 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:38.477913 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:38.478013 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:38.478018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.479109 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.479112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:38.479819 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:38.479830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.480431 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.480434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:38.480502 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:38.481146 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:38.481159 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:38.483635 (+ 2476us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106}
W20260929 16:26:38.484107 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:38.477355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41881 (local address 127.0.98.62:46605)
0929 16:26:38.477499 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:38.477503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:38.477522 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:26:38.477593 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:38.477596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:38.477646 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:38.477738 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:38.478122 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.478978 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.479941 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.480309 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.480421 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:38.483869 (+ 3448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41881: BlockingRecv error: recv got EOF from 127.0.98.1:41881 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 16:26:38.484107 9290 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:26:38.578187 9294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:38.578472 9294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:38.578517 9294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:38.582683 9294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:38.582754 9294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:38.582845 9294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:26:38.588433 9294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36411
--tserver_master_addrs=127.0.98.62:46605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:38.589617 9294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:38.590690 9294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:38.600607 9300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:38.600621 9299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:38.600735 9294 server_base.cc:1034] running on GCE node
W20260929 16:26:38.600621 9302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:38.601341 9294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:38.602005 9294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:38.603196 9294 hybrid_clock.cc:648] HybridClock initialized: now 1790699198603182 us; error 29 us; skew 500 ppm
Sep 29 16:26:38 dist-test-slave-vq9v krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699198, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:38.606215 9294 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 16:26:38.607244 9294 webserver.cc:460] Webserver started at http://127.0.98.2:36411/ using document root <none> and password file <none>
I20260929 16:26:38.607827 9294 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:38.607935 9294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:38.612092 9294 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:26:38.614468 9309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:38.615839 9294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:26:38.615975 9294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "c9f0e16b306c4ce8ab2441cfa4e48194"
format_stamp: "Formatted at 2026-09-29 16:26:37 on dist-test-slave-vq9v"
server_key: "0041c8f8e709df5b106468ea4c78197a"
server_key_iv: "ec847cc151e366df16d51f17d707bc5c"
server_key_version: "encryptionkey@0"
I20260929 16:26:38.616546 9294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:38.638240 9294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:38.641610 9294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:38.641832 9294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:38.642436 9294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:38.643318 9294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:38.643389 9294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:38.643465 9294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:38.643515 9294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:38.654911 9294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37981
I20260929 16:26:38.654994 9422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37981 every 8 connection(s)
I20260929 16:26:38.656005 9294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:26:38.657122 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 9294
I20260929 16:26:38.657639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46523
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46605
--webserver_port=34733
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:26:38 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:38 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:26:38.668159 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:38.658219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50167 (local address 127.0.98.62:46605)
0929 16:26:38.658384 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:38.658389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:38.659373 (+ 984us) server_negotiation.cc:407] Connection header received
0929 16:26:38.660736 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:38.660739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:38.660802 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:38.660912 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:38.662167 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.662690 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.663867 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.664095 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.664206 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:38.668034 (+ 3828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50167: BlockingRecv error: recv got EOF from 127.0.98.2:50167 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:26:38.668788 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:38.658698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.2:50167)
0929 16:26:38.659216 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:38.659259 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:38.660527 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:38.661104 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:38.661115 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:38.661543 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:38.662025 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:38.662038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.662814 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.662818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:38.663738 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:38.663747 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.664246 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.664249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:38.664430 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:38.665091 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:38.665107 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:38.667913 (+ 2806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":162,"threads_started":1}
W20260929 16:26:38.669318 9423 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46605 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:46605: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 16:26:38 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:38 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:38.675192 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:38.669762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.2:44725)
0929 16:26:38.669920 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:38.669933 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:38.670005 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:38.670230 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:38.670232 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:38.670406 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:38.670518 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:38.670523 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.671233 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.671237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:38.672146 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:38.672156 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.672615 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.672619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:38.672718 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:38.673181 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:38.673195 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:38.675044 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96}
W20260929 16:26:38.675275 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:38.669856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44725 (local address 127.0.98.62:46605)
0929 16:26:38.669964 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:38.669968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:38.669980 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:26:38.670086 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:38.670088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:38.670127 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:38.670194 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:38.670622 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.671099 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.672266 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.672513 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.672596 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:38.675160 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44725: BlockingRecv error: recv got EOF from 127.0.98.2:44725 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 29 16:26:38 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:38 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:38.681460 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:38.675799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.2:58265)
0929 16:26:38.675967 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:38.675984 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:38.676070 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:38.676276 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:38.676279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:38.676441 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:38.676528 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:38.676533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.677392 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.677395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:38.678257 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:38.678267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.678699 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.678704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:38.678801 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:38.679351 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:38.679369 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:38.681339 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91}
W20260929 16:26:38.681751 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:38.675891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58265 (local address 127.0.98.62:46605)
0929 16:26:38.675990 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:38.675994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:38.676031 (+ 37us) server_negotiation.cc:407] Connection header received
0929 16:26:38.676144 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:38.676146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:38.676180 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:38.676243 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:38.676678 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.677177 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.678391 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.678595 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.678682 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:38.681600 (+ 2918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58265: BlockingRecv error: recv got EOF from 127.0.98.2:58265 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260929 16:26:38.681730 9423 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:26:38.775008 9425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:38.775346 9425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:38.775426 9425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:38.779521 9425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:38.779634 9425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:38.779733 9425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:26:38.785073 9425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34733
--tserver_master_addrs=127.0.98.62:46605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:38.786348 9425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:38.787433 9425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:38.797394 9432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:38.797412 9434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:38.797403 9431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:38.798211 9425 server_base.cc:1034] running on GCE node
I20260929 16:26:38.798617 9425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:38.799268 9425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:38.800444 9425 hybrid_clock.cc:648] HybridClock initialized: now 1790699198800425 us; error 36 us; skew 500 ppm
Sep 29 16:26:38 dist-test-slave-vq9v krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699198, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:38.804217 9425 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 16:26:38.805235 9425 webserver.cc:460] Webserver started at http://127.0.98.3:34733/ using document root <none> and password file <none>
I20260929 16:26:38.805912 9425 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:38.805998 9425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:38.810655 9425 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:26:38.813129 9441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:38.814671 9425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:38.814812 9425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "84ae6e1ccf6146e8a39dbd0ad536b272"
format_stamp: "Formatted at 2026-09-29 16:26:37 on dist-test-slave-vq9v"
server_key: "077187cc603e27b019b2d9531b990cdd"
server_key_iv: "06a48189e80d7b971214e39c9cdea9d6"
server_key_version: "encryptionkey@0"
I20260929 16:26:38.815408 9425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:38.837145 9425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:38.840862 9425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:38.841102 9425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:38.841773 9425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:38.842641 9425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:38.842705 9425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:38.842772 9425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:38.842818 9425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:38.852172 9425 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46523
I20260929 16:26:38.852200 9554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46523 every 8 connection(s)
I20260929 16:26:38.853269 9425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:26:38.854463 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 9425
Sep 29 16:26:38 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:38 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:38.866035 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:38.855530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42538 (local address 127.0.98.62:46605)
0929 16:26:38.855807 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:38.855812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:38.855829 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:26:38.855915 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:38.855919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:38.855981 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:38.856129 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:38.857030 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.858024 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.858953 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.859328 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.859437 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:38.861382 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:38.861392 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:38.861413 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:38.861450 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:38.863531 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:38.864158 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:38.864164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:38.864172 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:38.864337 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:38.864630 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:38.864636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:38.864639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:38.865020 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:38.865257 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:38.865567 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 16:26:38.865816 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":101,"threads_started":1}
W20260929 16:26:38.866964 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:38.855222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46961 (local address 127.0.98.62:46605)
0929 16:26:38.855403 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:38.855411 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:38.856420 (+ 1009us) server_negotiation.cc:407] Connection header received
0929 16:26:38.858234 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:38.858245 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:38.858294 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:38.858378 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:38.860039 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.860942 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.862784 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.862999 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.863095 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:38.866855 (+ 3760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46961: BlockingRecv error: recv got EOF from 127.0.98.3:46961 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:26:38.867553 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:38.855729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.3:46961)
0929 16:26:38.856253 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:38.856316 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:38.858015 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:38.858576 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:38.858585 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:38.859073 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:38.859821 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:38.859844 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.861104 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.861107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:38.862632 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:38.862646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.863166 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.863170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:38.863378 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:38.864007 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:38.864024 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:38.866711 (+ 2687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":97,"threads_started":1}
W20260929 16:26:38.868055 9555 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46605 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:46605: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 16:26:38 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:38 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:38.873951 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:38.868545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.3:60115)
0929 16:26:38.868640 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:38.868655 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:38.868729 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:38.869008 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:38.869011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:38.869173 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:38.869313 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:38.869320 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.870083 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.870086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:38.870658 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:38.870665 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.871144 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.871147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:38.871231 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:38.871717 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:38.871729 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:38.873846 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 16:26:38.874104 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:38.868616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60115 (local address 127.0.98.62:46605)
0929 16:26:38.868767 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:38.868771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:38.868802 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:26:38.868842 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:38.868845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:38.868891 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:38.868979 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:38.869435 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.869979 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.870777 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.871020 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.871110 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:38.873979 (+ 2869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60115: BlockingRecv error: recv got EOF from 127.0.98.3:60115 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 16:26:38 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:38 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:38.879799 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:38.874486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.3:34175)
0929 16:26:38.874614 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:38.874626 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:38.874727 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:38.874978 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:38.874981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:38.875159 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:38.875235 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:38.875292 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.876123 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.876129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:38.876721 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:38.876729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.877248 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.877252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:38.877347 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:38.877800 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:38.877811 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:38.879669 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 16:26:38.879979 9292 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:38.874581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34175 (local address 127.0.98.62:46605)
0929 16:26:38.874761 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:38.874768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:38.874784 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:26:38.874826 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:38.874829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:38.874873 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:38.874962 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:38.875408 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.876007 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.876847 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:38.877090 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:38.877205 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:38.879829 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34175: BlockingRecv error: recv got EOF from 127.0.98.3:34175 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260929 16:26:38.880236 9555 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 16:26:39 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:39 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:39.491174 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:39.484840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.1:41301)
0929 16:26:39.485109 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:39.485123 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:39.485251 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:39.485524 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:39.485527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:39.485708 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:39.485847 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:39.485852 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:39.486744 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:39.486747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:39.487362 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:39.487391 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:39.487893 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:39.487896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:39.487974 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:39.488705 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:39.488718 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:39.491025 (+ 2307us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":93,"threads_started":1}
W20260929 16:26:39.491441 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:39.484916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41301 (local address 127.0.98.62:46605)
0929 16:26:39.485174 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:39.485177 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:39.485190 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:26:39.485358 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:39.485361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:39.485408 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:39.485487 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:39.485955 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:39.486639 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:39.487533 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:39.487763 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:39.487864 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:39.491165 (+ 3301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41301: BlockingRecv error: recv got EOF from 127.0.98.1:41301 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":114,"threads_started":1}
Sep 29 16:26:39 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:39 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:39.688179 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:39.682436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.2:46661)
0929 16:26:39.682717 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:39.682734 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:39.682847 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:39.683095 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:39.683097 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:39.683278 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:39.683384 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:39.683390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:39.684134 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:39.684137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:39.684707 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:39.684715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:39.685142 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:39.685144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:39.685232 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:39.685777 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:39.685805 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:39.687999 (+ 2194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":108,"threads_started":1}
W20260929 16:26:39.688474 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:39.682588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46661 (local address 127.0.98.62:46605)
0929 16:26:39.682725 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:39.682730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:39.682798 (+ 68us) server_negotiation.cc:407] Connection header received
0929 16:26:39.682943 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:39.682945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:39.682988 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:39.683055 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:39.683497 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:39.684030 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:39.684808 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:39.685036 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:39.685115 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:39.688159 (+ 3044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46661: BlockingRecv error: recv got EOF from 127.0.98.2:46661 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 16:26:39 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:39 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:39.886617 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:39.880986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.3:59417)
0929 16:26:39.881360 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:39.881379 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:39.881498 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:39.881755 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:39.881763 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:39.881982 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:39.882108 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:39.882115 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:39.882887 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:39.882890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:39.883462 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:39.883470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:39.883875 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:39.883877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:39.883940 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:39.884524 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:39.884536 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:39.886486 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":78,"threads_started":1}
W20260929 16:26:39.886735 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:39.881091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59417 (local address 127.0.98.62:46605)
0929 16:26:39.881252 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:39.881256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:39.881491 (+ 235us) server_negotiation.cc:407] Connection header received
0929 16:26:39.881578 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:39.881582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:39.881638 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:39.881706 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:39.882221 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:39.882777 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:39.883564 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:39.883782 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:39.883863 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:39.886620 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59417: BlockingRecv error: recv got EOF from 127.0.98.3:59417 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 16:26:40 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:40 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:40.497906 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:40.492123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.1:37071)
0929 16:26:40.492488 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:40.492503 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:40.492611 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:40.492914 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:40.492917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:40.493103 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:40.493234 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:40.493241 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:40.494040 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:40.494044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:40.494613 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:40.494622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:40.495109 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:40.495112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:40.495177 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:40.495784 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:40.495796 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:40.497762 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":113,"threads_started":1}
W20260929 16:26:40.498126 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:40.492270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37071 (local address 127.0.98.62:46605)
0929 16:26:40.492501 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:40.492505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:40.492581 (+ 76us) server_negotiation.cc:407] Connection header received
0929 16:26:40.492706 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:40.492708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:40.492753 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:40.492832 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:40.493359 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:40.493908 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:40.494746 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:40.494994 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:40.495089 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:40.497951 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37071: BlockingRecv error: recv got EOF from 127.0.98.1:37071 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":61,"threads_started":1}
Sep 29 16:26:40 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:40 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:40.694753 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:40.689073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.2:42171)
0929 16:26:40.689433 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:40.689454 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:40.689572 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:40.689836 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:40.689839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:40.690039 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:40.690137 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:40.690143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:40.690891 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:40.690894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:40.691490 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:40.691499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:40.692032 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:40.692035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:40.692099 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:40.692696 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:40.692716 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:40.694613 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":85,"threads_started":1}
W20260929 16:26:40.694916 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:40.689159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42171 (local address 127.0.98.62:46605)
0929 16:26:40.689440 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:40.689444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:40.689533 (+ 89us) server_negotiation.cc:407] Connection header received
0929 16:26:40.689655 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:40.689657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:40.689701 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:40.689773 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:40.690239 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:40.690767 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:40.691628 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:40.691886 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:40.692025 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:40.694746 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42171: BlockingRecv error: recv got EOF from 127.0.98.2:42171 (error 108)
Metrics: {"server-negotiator.queue_time_us":181}
Sep 29 16:26:40 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:40 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:40.893302 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:40.887519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.3:58037)
0929 16:26:40.887821 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:40.887838 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:40.887940 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:40.888233 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:40.888235 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:40.888416 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:40.888515 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:40.888521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:40.889447 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:40.889449 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:40.890031 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:40.890040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:40.890473 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:40.890476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:40.890535 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:40.891110 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:40.891122 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:40.893117 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":123,"threads_started":1}
W20260929 16:26:40.893460 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:40.887649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58037 (local address 127.0.98.62:46605)
0929 16:26:40.887891 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:40.887895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:40.887910 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:26:40.888028 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:40.888030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:40.888119 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:40.888199 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:40.888653 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:40.889345 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:40.890151 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:40.890367 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:40.890459 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:40.893306 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58037: BlockingRecv error: recv got EOF from 127.0.98.3:58037 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 29 16:26:41 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:41 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:41.504922 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:41.499094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.1:56247)
0929 16:26:41.499381 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:41.499401 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:41.499492 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:41.499832 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:41.499835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:41.500087 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:41.500189 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:41.500195 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:41.501018 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:41.501021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:41.501685 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:41.501695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:41.502120 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:41.502123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:41.502188 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:41.502747 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:41.502759 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:41.504731 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":101,"threads_started":1}
W20260929 16:26:41.504998 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:41.499317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56247 (local address 127.0.98.62:46605)
0929 16:26:41.499575 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:41.499580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:41.499596 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:26:41.499638 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:41.499641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:41.499693 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:41.499799 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:41.500280 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:41.500894 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:41.501792 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:41.502014 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:41.502095 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:41.504872 (+ 2777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56247: BlockingRecv error: recv got EOF from 127.0.98.1:56247 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":112,"threads_started":1}
Sep 29 16:26:41 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:41 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:41.701188 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:41.695516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.2:38901)
0929 16:26:41.695833 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:41.695849 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:41.696024 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:41.696291 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:41.696293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:41.696514 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:41.696607 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:41.696613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:41.697413 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:41.697418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:41.697986 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:41.697993 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:41.698441 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:41.698444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:41.698524 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:41.699080 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:41.699091 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:41.701041 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":87,"threads_started":1}
W20260929 16:26:41.701368 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:41.695650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38901 (local address 127.0.98.62:46605)
0929 16:26:41.695812 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:41.695817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:41.695939 (+ 122us) server_negotiation.cc:407] Connection header received
0929 16:26:41.696126 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:41.696130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:41.696181 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:41.696271 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:41.696727 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:41.697311 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:41.698107 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:41.698329 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:41.698436 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:41.701180 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38901: BlockingRecv error: recv got EOF from 127.0.98.2:38901 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 16:26:41 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:41 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:26:41.900743 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:41.894485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44741 (local address 127.0.98.62:46605)
0929 16:26:41.894612 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:41.894617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:41.894773 (+ 156us) server_negotiation.cc:407] Connection header received
0929 16:26:41.894964 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:41.894967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:41.895043 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:41.895122 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:41.895711 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:41.896346 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:41.897152 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:41.897468 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:41.897567 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:41.900649 (+ 3082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44741: BlockingRecv error: recv got EOF from 127.0.98.3:44741 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:26:41.900759 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:41.894294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.3:44741)
0929 16:26:41.894681 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:41.894694 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:41.894851 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:41.895178 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:41.895181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:41.895421 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:41.895564 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:41.895572 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:41.896456 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:41.896458 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:41.897029 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:41.897037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:41.897582 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:41.897585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:41.897650 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:41.898240 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:41.898254 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:41.900529 (+ 2275us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":175,"threads_started":1}
Sep 29 16:26:42 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:42 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:42.521713 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:42.505830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.1:33625)
0929 16:26:42.506088 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:42.506101 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:42.506190 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:42.506411 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:42.506413 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:42.506607 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:42.506702 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:42.506708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:42.507507 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:42.507510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:42.508059 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:42.508065 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:42.508465 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:42.508468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:42.508527 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:42.519395 (+ 10868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:42.519410 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:42.521588 (+ 2178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":80,"threads_started":1}
W20260929 16:26:42.521914 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:42.505881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33625 (local address 127.0.98.62:46605)
0929 16:26:42.506149 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:42.506153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:42.506165 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:26:42.506257 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:42.506259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:42.506299 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:42.506374 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:42.506850 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:42.507412 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:42.508156 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:42.508370 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:42.508446 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:42.521748 (+ 13302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33625: BlockingRecv error: recv got EOF from 127.0.98.1:33625 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":100,"threads_started":1}
Sep 29 16:26:42 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:42 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:42.708432 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:42.702112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.2:43933)
0929 16:26:42.702521 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:42.702542 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:42.702716 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:42.703044 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:42.703048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:42.703282 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:42.703391 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:42.703399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:42.704214 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:42.704218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:42.704795 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:42.704803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:42.705421 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:42.705424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:42.705495 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:42.706078 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:42.706095 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:42.708261 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":117,"threads_started":1}
W20260929 16:26:42.708505 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:42.702262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43933 (local address 127.0.98.62:46605)
0929 16:26:42.702521 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:42.702528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:42.702644 (+ 116us) server_negotiation.cc:407] Connection header received
0929 16:26:42.702858 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:42.702862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:42.702920 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:42.703026 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:42.703555 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:42.704083 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:42.704928 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:42.705176 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:42.705328 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:42.708397 (+ 3069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43933: BlockingRecv error: recv got EOF from 127.0.98.2:43933 (error 108)
Metrics: {"server-negotiator.queue_time_us":146}
Sep 29 16:26:42 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:42 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:42.907184 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:42.901739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.3:35539)
0929 16:26:42.902057 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:42.902073 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:42.902185 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:42.902476 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:42.902478 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:42.902654 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:42.902760 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:42.902765 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:42.903504 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:42.903507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:42.904082 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:42.904090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:42.904516 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:42.904519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:42.904581 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:42.905113 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:42.905125 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:42.907063 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":83,"threads_started":1}
W20260929 16:26:42.907337 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:42.901926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35539 (local address 127.0.98.62:46605)
0929 16:26:42.902076 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:42.902079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:42.902145 (+ 66us) server_negotiation.cc:407] Connection header received
0929 16:26:42.902268 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:42.902270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:42.902315 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:42.902385 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:42.902871 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:42.903383 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:42.904190 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:42.904401 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:42.904479 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:42.907212 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35539: BlockingRecv error: recv got EOF from 127.0.98.3:35539 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 16:26:43 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:43 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:43.528154 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:43.522449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.1:42631)
0929 16:26:43.522766 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:43.522780 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:43.522897 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:43.523143 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:43.523145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:43.523322 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:43.523421 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:43.523426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:43.524275 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:43.524279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:43.524893 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:43.524904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:43.525551 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:43.525554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:43.525624 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:43.526175 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:43.526188 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:43.528024 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":88,"threads_started":1}
W20260929 16:26:43.528307 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:43.522565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42631 (local address 127.0.98.62:46605)
0929 16:26:43.522790 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:43.522793 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:43.522857 (+ 64us) server_negotiation.cc:407] Connection header received
0929 16:26:43.522982 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:43.522984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:43.523032 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:43.523107 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:43.523587 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:43.524143 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:43.525023 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:43.525413 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:43.525536 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:43.528173 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42631: BlockingRecv error: recv got EOF from 127.0.98.1:42631 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":55,"threads_started":1}
Sep 29 16:26:43 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:43 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:43.715386 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:43.709320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.2:45109)
0929 16:26:43.709592 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:43.709606 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:43.709703 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:43.710019 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:43.710021 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:43.710201 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:43.710296 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:43.710301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:43.711073 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:43.711077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:43.711740 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:43.711752 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:43.712226 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:43.712230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:43.712317 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:43.712882 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:43.712916 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:43.715228 (+ 2312us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":98,"threads_started":1}
W20260929 16:26:43.715776 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:43.709485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45109 (local address 127.0.98.62:46605)
0929 16:26:43.709626 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:43.709629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:43.709666 (+ 37us) server_negotiation.cc:407] Connection header received
0929 16:26:43.709830 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:43.709833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:43.709883 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:43.709966 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:43.710419 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:43.710952 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:43.711865 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:43.712106 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:43.712218 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:43.715676 (+ 3458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45109: BlockingRecv error: recv got EOF from 127.0.98.2:45109 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 16:26:43 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:43 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:43.914026 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:43.908118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.3:39343)
0929 16:26:43.908533 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:43.908550 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:43.908682 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:43.908960 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:43.908962 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:43.909169 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:43.909288 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:43.909294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:43.910076 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:43.910082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:43.910657 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:43.910664 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:43.911113 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:43.911116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:43.911201 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:43.911787 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:43.911798 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:43.913877 (+ 2079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":87,"threads_started":1}
W20260929 16:26:43.914225 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:43.908260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39343 (local address 127.0.98.62:46605)
0929 16:26:43.908520 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:43.908524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:43.908630 (+ 106us) server_negotiation.cc:407] Connection header received
0929 16:26:43.908787 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:43.908794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:43.908847 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:43.908938 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:43.909399 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:43.909948 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:43.910772 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:43.911002 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:43.911110 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:43.914080 (+ 2970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39343: BlockingRecv error: recv got EOF from 127.0.98.3:39343 (error 108)
Metrics: {"server-negotiator.queue_time_us":163}
Sep 29 16:26:44 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:44 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:44.536243 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:44.529036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.1:44107)
0929 16:26:44.529398 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:44.529411 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:44.529506 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:44.529800 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:44.529802 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:44.529992 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:44.530086 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:44.530092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:44.531367 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:44.531370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:44.531966 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:44.532010 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:44.532840 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:44.532847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:44.533040 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:44.533767 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:44.533780 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:44.536000 (+ 2220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":80,"threads_started":1}
W20260929 16:26:44.536270 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:44.529161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44107 (local address 127.0.98.62:46605)
0929 16:26:44.529455 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:44.529462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:44.529481 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:26:44.529587 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:44.529592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:44.529656 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:44.529779 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:44.530201 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:44.531205 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:44.532192 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:44.532651 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:44.532769 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:44.536160 (+ 3391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44107: BlockingRecv error: recv got EOF from 127.0.98.1:44107 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":93,"threads_started":1}
Sep 29 16:26:44 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:44 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:44.722416 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:44.716234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.2:46231)
0929 16:26:44.716476 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:44.716495 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:44.716645 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:44.716948 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:44.716954 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:44.717237 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:44.717339 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:44.717345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:44.718124 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:44.718127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:44.718711 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:44.718720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:44.719179 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:44.719185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:44.719268 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:44.719959 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:44.719971 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:44.722292 (+ 2321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":80,"threads_started":1}
W20260929 16:26:44.722620 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:44.716296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46231 (local address 127.0.98.62:46605)
0929 16:26:44.716448 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:44.716453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:44.716603 (+ 150us) server_negotiation.cc:407] Connection header received
0929 16:26:44.716740 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:44.716745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:44.716800 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:44.716880 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:44.717471 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:44.718006 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:44.718814 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:44.719027 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:44.719103 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:44.722479 (+ 3376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46231: BlockingRecv error: recv got EOF from 127.0.98.2:46231 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 16:26:44 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:44 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:44.921361 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:44.915007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.3:60111)
0929 16:26:44.915453 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:44.915476 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:44.915671 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:44.915994 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:44.915996 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:44.916213 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:44.916324 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:44.916330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:44.917141 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:44.917148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:44.917904 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:44.917913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:44.918418 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:44.918422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:44.918517 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:44.919048 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:44.919060 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:44.921132 (+ 2072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":96,"threads_started":1}
W20260929 16:26:44.921414 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:44.915156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60111 (local address 127.0.98.62:46605)
0929 16:26:44.915428 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:44.915433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:44.915559 (+ 126us) server_negotiation.cc:407] Connection header received
0929 16:26:44.915804 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:44.915807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:44.915856 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:44.915964 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:44.916495 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:44.917023 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:44.918063 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:44.918295 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:44.918416 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:44.921301 (+ 2885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60111: BlockingRecv error: recv got EOF from 127.0.98.3:60111 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
Sep 29 16:26:45 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:45 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:45.543638 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:45.537376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.1:48619)
0929 16:26:45.537724 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:45.537738 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:45.537864 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:45.538147 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:45.538149 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:45.538329 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:45.538432 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:45.538437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:45.539206 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:45.539209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:45.539853 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:45.539864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:45.540435 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:45.540439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:45.540524 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:45.541168 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:45.541182 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:45.543514 (+ 2332us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":92,"threads_started":1}
W20260929 16:26:45.543777 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:45.537527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48619 (local address 127.0.98.62:46605)
0929 16:26:45.537768 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:45.537772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:45.537790 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:26:45.537984 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:45.537986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:45.538030 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:45.538110 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:45.538544 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:45.539094 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:45.540025 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:45.540300 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:45.540381 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:45.543645 (+ 3264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48619: BlockingRecv error: recv got EOF from 127.0.98.1:48619 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":68,"threads_started":1}
Sep 29 16:26:45 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:45 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:45.729398 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:45.723220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.2:41153)
0929 16:26:45.723617 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:45.723632 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:45.723778 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:45.724072 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:45.724075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:45.724288 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:45.724387 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:45.724393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:45.725167 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:45.725172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:45.725835 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:45.725845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:45.726344 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:45.726347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:45.726430 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:45.727026 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:45.727038 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:45.729202 (+ 2164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":99,"threads_started":1}
W20260929 16:26:45.729561 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:45.723353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41153 (local address 127.0.98.62:46605)
0929 16:26:45.723579 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:45.723583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:45.723727 (+ 144us) server_negotiation.cc:407] Connection header received
0929 16:26:45.723888 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:45.723891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:45.723955 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:45.724046 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:45.724513 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:45.725052 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:45.725989 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:45.726222 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:45.726339 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:45.729452 (+ 3113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41153: BlockingRecv error: recv got EOF from 127.0.98.2:41153 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 29 16:26:45 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:45 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:45.929281 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:45.922243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.3:40831)
0929 16:26:45.922544 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:45.922558 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:45.922657 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:45.922962 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:45.922964 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:45.923152 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:45.923250 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:45.923255 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:45.924500 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:45.924504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:45.925107 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:45.925117 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:45.925894 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:45.925897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:45.925969 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:45.926603 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:45.926614 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:45.929026 (+ 2412us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":86,"threads_started":1}
W20260929 16:26:45.929297 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:45.922347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40831 (local address 127.0.98.62:46605)
0929 16:26:45.922523 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:45.922529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:45.922643 (+ 114us) server_negotiation.cc:407] Connection header received
0929 16:26:45.922745 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:45.922751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:45.922825 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:45.922936 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:45.923376 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:45.924348 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:45.925294 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:45.925749 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:45.925879 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:45.929147 (+ 3268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40831: BlockingRecv error: recv got EOF from 127.0.98.3:40831 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 16:26:46 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:46 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:46.550388 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:46.544528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.1:34815)
0929 16:26:46.544872 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:46.544890 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:46.544999 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:46.545289 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:46.545292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:46.545505 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:46.545626 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:46.545633 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:46.546432 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:46.546435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:46.547083 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:46.547092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:46.547606 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:46.547609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:46.547683 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:46.548211 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:46.548223 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:46.550258 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":93,"threads_started":1}
W20260929 16:26:46.550570 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:46.544695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34815 (local address 127.0.98.62:46605)
0929 16:26:46.544948 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:46.544952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:46.544966 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:26:46.545081 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:46.545083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:46.545122 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:46.545238 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:46.545759 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:46.546321 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:46.547227 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:46.547472 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:46.547568 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:46.550404 (+ 2836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34815: BlockingRecv error: recv got EOF from 127.0.98.1:34815 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":94,"threads_started":1}
Sep 29 16:26:46 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:46 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:46.737272 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:46.730498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.2:41841)
0929 16:26:46.730789 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:46.730807 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:46.730969 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:46.731245 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:46.731247 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:46.731428 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:46.731606 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:46.731616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:46.732682 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:46.732689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:46.733634 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:46.733646 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:46.734381 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:46.734385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:46.734514 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:46.735165 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:46.735179 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:46.737087 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":116,"threads_started":1}
W20260929 16:26:46.737555 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:46.730631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41841 (local address 127.0.98.62:46605)
0929 16:26:46.730789 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:46.730796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:46.730897 (+ 101us) server_negotiation.cc:407] Connection header received
0929 16:26:46.731084 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:46.731086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:46.731137 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:46.731225 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:46.731731 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:46.732561 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:46.733774 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:46.734101 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:46.734200 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:46.737349 (+ 3149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41841: BlockingRecv error: recv got EOF from 127.0.98.2:41841 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 16:26:46 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:46 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:46.936455 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:46.930399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.3:46889)
0929 16:26:46.930814 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:46.930827 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:46.930990 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:46.931282 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:46.931285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:46.931513 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:46.931614 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:46.931620 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:46.932454 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:46.932460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:46.933035 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:46.933043 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:46.933613 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:46.933617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:46.933715 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:46.934240 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:46.934252 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:46.936279 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":201,"threads_started":1}
W20260929 16:26:46.936605 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:46.930479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46889 (local address 127.0.98.62:46605)
0929 16:26:46.930730 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:46.930735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:46.930919 (+ 184us) server_negotiation.cc:407] Connection header received
0929 16:26:46.931103 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:46.931107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:46.931163 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:46.931259 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:46.931747 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:46.932296 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:46.933162 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:46.933459 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:46.933570 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:46.936487 (+ 2917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46889: BlockingRecv error: recv got EOF from 127.0.98.3:46889 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
Sep 29 16:26:47 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:47 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:47.564306 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:47.551307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.1:37699)
0929 16:26:47.551582 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:47.551598 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:47.551707 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:47.552059 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:47.552062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:47.552326 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:47.552450 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:47.552458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:47.553380 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:47.553382 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:47.553940 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:47.553948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:47.554439 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:47.554442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:47.554521 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:47.561969 (+ 7448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:47.561987 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:47.564141 (+ 2154us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":103,"threads_started":1}
W20260929 16:26:47.564427 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:47.551408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37699 (local address 127.0.98.62:46605)
0929 16:26:47.551642 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:47.551645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:47.551660 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:26:47.551834 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:47.551837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:47.551894 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:47.551995 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:47.552578 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:47.553280 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:47.554058 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:47.554275 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:47.554351 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:47.564298 (+ 9947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37699: BlockingRecv error: recv got EOF from 127.0.98.1:37699 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":85,"threads_started":1}
Sep 29 16:26:47 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:47 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:47.744361 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:47.738355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.2:46655)
0929 16:26:47.738775 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:47.738793 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:47.738958 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:47.739274 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:47.739277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:47.739488 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:47.739595 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:47.739601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:47.740372 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:47.740378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:47.740951 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:47.740960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:47.741533 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:47.741537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:47.741642 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:47.742172 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:47.742184 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:47.744218 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":145,"threads_started":1}
W20260929 16:26:47.744724 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:47.738611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46655 (local address 127.0.98.62:46605)
0929 16:26:47.738777 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:47.738785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:47.738893 (+ 108us) server_negotiation.cc:407] Connection header received
0929 16:26:47.739106 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:47.739112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:47.739157 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:47.739246 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:47.739723 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:47.740258 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:47.741091 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:47.741411 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:47.741525 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:47.744428 (+ 2903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46655: BlockingRecv error: recv got EOF from 127.0.98.2:46655 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 29 16:26:47 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:47 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:47.942879 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:47.937445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.3:60139)
0929 16:26:47.937776 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:47.937789 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:47.937887 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:47.938118 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:47.938120 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:47.938296 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:47.938393 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:47.938398 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:47.939178 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:47.939181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:47.939760 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:47.939769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:47.940200 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:47.940202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:47.940262 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:47.940774 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:47.940786 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:47.942760 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":129,"threads_started":1}
W20260929 16:26:47.943117 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:47.937496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60139 (local address 127.0.98.62:46605)
0929 16:26:47.937628 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:47.937631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:47.937858 (+ 227us) server_negotiation.cc:407] Connection header received
0929 16:26:47.937958 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:47.937960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:47.938009 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:47.938076 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:47.938538 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:47.939061 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:47.939873 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:47.940096 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:47.940182 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:47.942992 (+ 2810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60139: BlockingRecv error: recv got EOF from 127.0.98.3:60139 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 16:26:48 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:48 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:48.570779 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:48.565336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.1:39061)
0929 16:26:48.565669 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:48.565684 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:48.565778 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:48.566039 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:48.566041 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:48.566223 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:48.566316 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:48.566320 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:48.567191 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:48.567193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:48.567758 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:48.567766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:48.568248 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:48.568251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:48.568312 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:48.568808 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:48.568819 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:48.570660 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":82,"threads_started":1}
W20260929 16:26:48.571012 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:48.565480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39061 (local address 127.0.98.62:46605)
0929 16:26:48.565766 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:48.565770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:48.565789 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:26:48.565864 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:48.565867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:48.565914 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:48.566012 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:48.566484 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:48.567081 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:48.567905 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:48.568136 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:48.568226 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:48.570849 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39061: BlockingRecv error: recv got EOF from 127.0.98.1:39061 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
Sep 29 16:26:48 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:48 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:48.751103 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:48.745341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.2:42487)
0929 16:26:48.745696 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:48.745710 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:48.745812 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:48.746068 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:48.746071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:48.746273 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:48.746368 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:48.746373 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:48.747174 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:48.747176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:48.747753 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:48.747762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:48.748261 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:48.748263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:48.748322 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:48.748918 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:48.748930 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:48.750981 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":168,"threads_started":1}
W20260929 16:26:48.751305 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:48.745566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42487 (local address 127.0.98.62:46605)
0929 16:26:48.745704 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:48.745715 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:48.745782 (+ 67us) server_negotiation.cc:407] Connection header received
0929 16:26:48.745898 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:48.745901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:48.745947 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:48.746028 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:48.746511 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:48.747069 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:48.747893 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:48.748153 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:48.748256 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:48.751167 (+ 2911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42487: BlockingRecv error: recv got EOF from 127.0.98.2:42487 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 16:26:48 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:48 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:48.949815 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:48.943887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.3:35881)
0929 16:26:48.944203 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:48.944230 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:48.944344 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:48.944647 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:48.944650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:48.944885 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:48.945003 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:48.945013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:48.945944 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:48.945949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:48.946562 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:48.946571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:48.947045 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:48.947049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:48.947136 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:48.947732 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:48.947743 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:48.949678 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":139,"threads_started":1}
W20260929 16:26:48.949992 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:48.944136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35881 (local address 127.0.98.62:46605)
0929 16:26:48.944310 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:48.944315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:48.944332 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:26:48.944473 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:48.944475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:48.944534 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:48.944625 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:48.945143 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:48.945834 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:48.946692 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:48.946922 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:48.947041 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:48.949826 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35881: BlockingRecv error: recv got EOF from 127.0.98.3:35881 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 16:26:49 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:49 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:49.577580 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:49.571596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.1:36319)
0929 16:26:49.571887 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:49.571905 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:49.572004 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:49.572327 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:49.572330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:49.572610 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:49.572725 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:49.572731 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:49.573596 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:49.573600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:49.574161 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:49.574169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:49.574686 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:49.574689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:49.574757 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:49.575279 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:49.575291 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:49.577455 (+ 2164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":78,"threads_started":1}
W20260929 16:26:49.577786 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:49.571699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36319 (local address 127.0.98.62:46605)
0929 16:26:49.571973 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:49.571977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:49.571994 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:26:49.572113 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:49.572119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:49.572175 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:49.572280 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:49.572831 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:49.573475 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:49.574312 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:49.574554 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:49.574647 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:49.577605 (+ 2958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36319: BlockingRecv error: recv got EOF from 127.0.98.1:36319 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":73,"threads_started":1}
Sep 29 16:26:49 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:49 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:49.757439 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:49.751871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.2:60827)
0929 16:26:49.752183 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:49.752197 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:49.752286 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:49.752595 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:49.752598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:49.752779 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:49.752877 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:49.752882 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:49.753710 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:49.753713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:49.754296 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:49.754305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:49.754770 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:49.754774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:49.754848 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:49.755389 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:49.755403 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:49.757316 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":120,"threads_started":1}
W20260929 16:26:49.757645 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:49.752024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60827 (local address 127.0.98.62:46605)
0929 16:26:49.752212 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:49.752221 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:49.752244 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:26:49.752416 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:49.752419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:49.752469 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:49.752577 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:49.753002 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:49.753595 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:49.754403 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:49.754639 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:49.754717 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:49.757508 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60827: BlockingRecv error: recv got EOF from 127.0.98.2:60827 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 16:26:49 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:49 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:49.957021 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:49.950780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.3:45527)
0929 16:26:49.951043 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:49.951060 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:49.951171 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:49.951450 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:49.951458 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:49.951673 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:49.951798 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:49.951807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:49.952814 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:49.952817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:49.953422 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:49.953430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:49.953912 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:49.953916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:49.953992 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:49.954622 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:49.954635 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:49.956898 (+ 2263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":90,"threads_started":1}
W20260929 16:26:49.957320 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:49.950871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45527 (local address 127.0.98.62:46605)
0929 16:26:49.951022 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:49.951027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:49.951148 (+ 121us) server_negotiation.cc:407] Connection header received
0929 16:26:49.951250 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:49.951253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:49.951316 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:49.951393 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:49.951925 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:49.952712 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:49.953539 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:49.953760 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:49.953840 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:49.957127 (+ 3287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45527: BlockingRecv error: recv got EOF from 127.0.98.3:45527 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 16:26:50 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:50 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:50.585253 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:50.578450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.1:58605)
0929 16:26:50.578748 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:50.578761 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:50.578880 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:50.579282 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:50.579284 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:50.579459 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:50.579554 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:50.579559 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:50.580768 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:50.580771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:50.581414 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:50.581424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:50.581928 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:50.581932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:50.582020 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:50.582656 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:50.582667 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:50.584968 (+ 2301us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":102,"threads_started":1}
W20260929 16:26:50.585249 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:50.578477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58605 (local address 127.0.98.62:46605)
0929 16:26:50.579032 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:50.579038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:50.579055 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:26:50.579100 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:50.579106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:50.579164 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:50.579266 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:50.579669 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:50.580645 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:50.581536 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:50.581807 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:50.581943 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:50.585095 (+ 3152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58605: BlockingRecv error: recv got EOF from 127.0.98.1:58605 (error 108)
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":101,"threads_started":1}
Sep 29 16:26:50 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:50 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:50.763402 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:50.758194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.2:39089)
0929 16:26:50.758482 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:50.758496 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:50.758598 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:50.758801 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:50.758803 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:50.758986 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:50.759083 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:50.759108 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:50.759871 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:50.759874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:50.760450 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:50.760460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:50.760901 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:50.760904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:50.760966 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:50.761512 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:50.761524 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:50.763286 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":65,"threads_started":1}
W20260929 16:26:50.763514 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:50.758324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39089 (local address 127.0.98.62:46605)
0929 16:26:50.758476 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:50.758480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:50.758567 (+ 87us) server_negotiation.cc:407] Connection header received
0929 16:26:50.758676 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:50.758678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:50.758722 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:50.758788 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:50.759202 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:50.759741 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:50.760588 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:50.760800 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:50.760878 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:50.763416 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39089: BlockingRecv error: recv got EOF from 127.0.98.2:39089 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 16:26:50 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:50 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:50.964310 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:50.957988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.3:45377)
0929 16:26:50.958343 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:50.958360 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:50.958519 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:50.958809 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:50.958812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:50.959028 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:50.959120 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:50.959126 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:50.959879 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:50.959884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:50.960571 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:50.960582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:50.961294 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:50.961298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:50.961393 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:50.962043 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:50.962054 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:50.964148 (+ 2094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":112,"threads_started":1}
W20260929 16:26:50.964406 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:50.958162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45377 (local address 127.0.98.62:46605)
0929 16:26:50.958334 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:50.958339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:50.958453 (+ 114us) server_negotiation.cc:407] Connection header received
0929 16:26:50.958626 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:50.958629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:50.958693 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:50.958788 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:50.959241 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:50.959766 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:50.960746 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:50.961125 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:50.961290 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:50.964279 (+ 2989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45377: BlockingRecv error: recv got EOF from 127.0.98.3:45377 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 16:26:51 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:51 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:51.592643 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:51.586125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.1:32825)
0929 16:26:51.586472 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:51.586490 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:51.586636 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:51.587004 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:51.587010 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:51.587291 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:51.587418 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:51.587426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:51.588368 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:51.588371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:51.589144 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:51.589154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:51.589788 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:51.589791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:51.589861 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:51.590421 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:51.590434 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:51.592497 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":113,"threads_started":1}
W20260929 16:26:51.592803 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:51.586255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32825 (local address 127.0.98.62:46605)
0929 16:26:51.586536 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:51.586539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:51.586554 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:26:51.586757 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:51.586760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:51.586822 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:51.586935 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:51.587587 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:51.588238 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:51.589451 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:51.589679 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:51.589764 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:51.592653 (+ 2889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:32825: BlockingRecv error: recv got EOF from 127.0.98.1:32825 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":74,"threads_started":1}
Sep 29 16:26:51 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:51 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:51.770339 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:51.764623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.2:45531)
0929 16:26:51.764987 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:51.765004 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:51.765156 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:51.765467 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:51.765470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:51.765677 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:51.765774 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:51.765779 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:51.766563 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:51.766567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:51.767131 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:51.767138 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:51.767592 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:51.767596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:51.767681 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:51.768211 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:51.768222 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:51.770168 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":108,"threads_started":1}
W20260929 16:26:51.770392 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:51.764797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45531 (local address 127.0.98.62:46605)
0929 16:26:51.764988 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:51.764995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:51.765090 (+ 95us) server_negotiation.cc:407] Connection header received
0929 16:26:51.765290 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:51.765293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:51.765350 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:51.765441 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:51.765891 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:51.766431 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:51.767249 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:51.767479 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:51.767589 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:51.770285 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45531: BlockingRecv error: recv got EOF from 127.0.98.2:45531 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 16:26:51 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:51 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:51.971267 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:51.965150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.3:43117)
0929 16:26:51.965496 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:51.965554 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:51.965711 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:51.965987 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:51.965990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:51.966201 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:51.966290 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:51.966296 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:51.967050 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:51.967056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:51.967621 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:51.967628 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:51.968095 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:51.968099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:51.968182 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:51.968773 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:51.968787 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:51.971094 (+ 2307us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":139,"threads_started":1}
W20260929 16:26:51.971312 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:51.965390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43117 (local address 127.0.98.62:46605)
0929 16:26:51.965539 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:51.965544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:51.965645 (+ 101us) server_negotiation.cc:407] Connection header received
0929 16:26:51.965827 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:51.965832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:51.965883 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:51.965969 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:51.966409 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:51.966932 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:51.967738 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:51.967973 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:51.968090 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:51.971212 (+ 3122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43117: BlockingRecv error: recv got EOF from 127.0.98.3:43117 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 16:26:52 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:52 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:52.603901 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:52.593573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.1:46039)
0929 16:26:52.593874 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:52.593889 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:52.594004 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:52.594288 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:52.594290 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:52.594563 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:52.594684 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:52.594693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:52.595851 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:52.595855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:52.596618 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:52.596628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:52.597113 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:52.597117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:52.597265 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:52.601703 (+ 4438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:52.601719 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:52.603766 (+ 2047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":113,"threads_started":1}
W20260929 16:26:52.604115 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:52.593729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46039 (local address 127.0.98.62:46605)
0929 16:26:52.594043 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:52.594047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:52.594062 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:26:52.594105 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:52.594107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:52.594150 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:52.594247 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:52.594842 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:52.595697 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:52.596741 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:52.596962 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:52.597055 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:52.603949 (+ 6894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46039: BlockingRecv error: recv got EOF from 127.0.98.1:46039 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":91,"threads_started":1}
Sep 29 16:26:52 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:52 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:52.777899 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:52.771258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.2:37971)
0929 16:26:52.771677 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:52.771699 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:52.771840 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:52.772119 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:52.772122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:52.772343 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:52.772436 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:52.772442 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:52.773243 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:52.773249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:52.773830 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:52.773838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:52.774372 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:52.774376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:52.774461 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:52.775191 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:52.775209 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:52.777685 (+ 2476us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":116,"threads_started":1}
W20260929 16:26:52.777974 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:52.771329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37971 (local address 127.0.98.62:46605)
0929 16:26:52.771637 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:52.771642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:52.771779 (+ 137us) server_negotiation.cc:407] Connection header received
0929 16:26:52.771941 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:52.771945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:52.771998 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:52.772092 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:52.772582 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:52.773102 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:52.773959 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:52.774266 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:52.774367 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:52.777835 (+ 3468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37971: BlockingRecv error: recv got EOF from 127.0.98.2:37971 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 16:26:52 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:52 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:52.978026 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:52.972089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.3:46039)
0929 16:26:52.972395 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:52.972409 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:52.972512 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:52.972734 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:52.972736 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:52.972910 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:52.973004 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:52.973009 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:52.973860 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:52.973863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:52.974438 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:52.974446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:52.974858 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:52.974861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:52.974924 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:52.975513 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:52.975525 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:52.977859 (+ 2334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":87,"threads_started":1}
W20260929 16:26:52.978189 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:52.972191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46039 (local address 127.0.98.62:46605)
0929 16:26:52.972376 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:52.972380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:52.972482 (+ 102us) server_negotiation.cc:407] Connection header received
0929 16:26:52.972581 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:52.972584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:52.972629 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:52.972697 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:52.973148 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:52.973760 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:52.974541 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:52.974759 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:52.974837 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:52.978023 (+ 3186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46039: BlockingRecv error: recv got EOF from 127.0.98.3:46039 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 29 16:26:53 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:53 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:53.610751 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:53.604894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.1:39023)
0929 16:26:53.605189 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:53.605251 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:53.605356 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:53.605609 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:53.605611 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:53.605797 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:53.605895 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:53.605901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:53.606653 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:53.606656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:53.607282 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:53.607290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:53.607726 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:53.607729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:53.607798 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:53.608430 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:53.608443 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:53.610622 (+ 2179us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":119,"threads_started":1}
W20260929 16:26:53.610935 9617 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:53.604999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39023 (local address 127.0.98.62:46605)
0929 16:26:53.605263 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:53.605266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:53.605311 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:26:53.605425 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:53.605427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:53.605468 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:53.605549 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:53.606000 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:53.606519 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:53.607376 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:53.607617 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:53.607698 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:53.610777 (+ 3079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39023: BlockingRecv error: recv got EOF from 127.0.98.1:39023 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":102,"threads_started":1}
Sep 29 16:26:53 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:26:53 dist-test-slave-vq9v krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:26:53.785226 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:53.778922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46605 (local address 127.0.98.2:42767)
0929 16:26:53.779290 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:53.779303 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:53.779401 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:53.779669 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:53.779672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:53.779845 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:53.779950 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:53.779955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:53.780730 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:53.780733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:53.781406 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:53.781416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:53.782011 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:53.782015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:53.782122 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:53.782753 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:53.782765 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:53.785023 (+ 2258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46605: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":110,"threads_started":1}
W20260929 16:26:53.785318 9617 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:53.779074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42767 (local address 127.0.98.62:46605)
0929 16:26:53.779219 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:53.779223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:53.779375 (+ 152us) server_negotiation.cc:407] Connection header received
0929 16:26:53.779484 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:53.779486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:53.779529 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:53.779614 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:53.780075 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:53.780607 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:53.781546 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:53.781811 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:53.781914 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:53.785163 (+ 3249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42767: BlockingRecv error: recv got EOF from 127.0.98.2:42767 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:26:53.864897 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 9142
I20260929 16:26:53.870885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 9294
I20260929 16:26:53.876515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 9425
I20260929 16:26:53.882361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 9072
2026-09-29T16:26:53Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18097 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T16:26:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:26:53Z Disabled control of system clock
I20260929 16:26:53.914860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43935
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 16:26:54.031003 9622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:54.031291 9622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:54.031325 9622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:54.035410 9622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:26:54.035482 9622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:54.035497 9622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:54.035514 9622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:26:54.035529 9622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:26:54.040983 9622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43935
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:54.042137 9622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:54.043464 9622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:54.052474 9630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:54.052525 9627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:54.052637 9622 server_base.cc:1034] running on GCE node
W20260929 16:26:54.052539 9628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:54.053520 9622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:54.054762 9622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:54.055972 9622 hybrid_clock.cc:648] HybridClock initialized: now 1790699214055945 us; error 46 us; skew 500 ppm
I20260929 16:26:54.058113 9622 webserver.cc:460] Webserver started at http://127.0.98.62:34083/ using document root <none> and password file <none>
I20260929 16:26:54.058782 9622 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:54.058851 9622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:54.059077 9622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:54.061318 9622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "2fbfe828d7454849ad7041b61b8a3866"
format_stamp: "Formatted at 2026-09-29 16:26:54 on dist-test-slave-vq9v"
server_key: "4919e19ffc1ddd605ebc243db9c143c3"
server_key_iv: "060ea6d1c3b93e755c5da77e106ed191"
server_key_version: "encryptionkey@0"
I20260929 16:26:54.061944 9622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "2fbfe828d7454849ad7041b61b8a3866"
format_stamp: "Formatted at 2026-09-29 16:26:54 on dist-test-slave-vq9v"
server_key: "4919e19ffc1ddd605ebc243db9c143c3"
server_key_iv: "060ea6d1c3b93e755c5da77e106ed191"
server_key_version: "encryptionkey@0"
I20260929 16:26:54.066089 9622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 16:26:54.068894 9636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:54.070412 9622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:26:54.070632 9622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "2fbfe828d7454849ad7041b61b8a3866"
format_stamp: "Formatted at 2026-09-29 16:26:54 on dist-test-slave-vq9v"
server_key: "4919e19ffc1ddd605ebc243db9c143c3"
server_key_iv: "060ea6d1c3b93e755c5da77e106ed191"
server_key_version: "encryptionkey@0"
I20260929 16:26:54.070773 9622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:54.082026 9622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:54.082795 9622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:54.083024 9622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:54.090847 9622 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43935
I20260929 16:26:54.090853 9688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43935 every 8 connection(s)
I20260929 16:26:54.091849 9622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:26:54.095168 9689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:54.101063 9689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866: Bootstrap starting.
I20260929 16:26:54.101560 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 9622
I20260929 16:26:54.101701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:26:54.102015 392 external_mini_cluster.cc:1428] Setting key 6333cbb5d637f74a74960e1793eb69e9
I20260929 16:26:54.103911 9689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:54.104694 9689 log.cc:824] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:54.107172 9689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866: No bootstrap required, opened a new log
I20260929 16:26:54.111084 9689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbfe828d7454849ad7041b61b8a3866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43935 } }
I20260929 16:26:54.111325 9689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:54.111406 9689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fbfe828d7454849ad7041b61b8a3866, State: Initialized, Role: FOLLOWER
I20260929 16:26:54.111856 9689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbfe828d7454849ad7041b61b8a3866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43935 } }
I20260929 16:26:54.111979 9689 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:26:54.112005 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:54.103519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43028 (local address 127.0.98.62:43935)
0929 16:26:54.104211 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:54.104226 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:54.104264 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:26:54.105018 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:54.105039 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:54.105544 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:54.105908 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:54.106283 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:54.107318 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:54.108254 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:54.108750 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:54.108938 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:54.109913 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:54.109918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:54.109932 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:54.110366 (+ 434us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:54.110427 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:54.110449 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:54.110564 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:54.110725 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 16:26:54.111008 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":167,"threads_started":1}
I20260929 16:26:54.112073 9689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:26:54.112187 9689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:54.113318 9689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbfe828d7454849ad7041b61b8a3866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43935 } }
I20260929 16:26:54.113720 9689 leader_election.cc:302] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fbfe828d7454849ad7041b61b8a3866; no voters:
I20260929 16:26:54.114104 9689 leader_election.cc:288] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:26:54.114185 9694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:54.114398 9694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866 [term 1 LEADER]: Becoming Leader. State: Replica: 2fbfe828d7454849ad7041b61b8a3866, State: Running, Role: LEADER
I20260929 16:26:54.114737 9694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbfe828d7454849ad7041b61b8a3866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43935 } }
I20260929 16:26:54.115428 9689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:26:54.116652 9695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fbfe828d7454849ad7041b61b8a3866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbfe828d7454849ad7041b61b8a3866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43935 } } }
I20260929 16:26:54.116793 9695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:54.117043 9696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fbfe828d7454849ad7041b61b8a3866. Latest consensus state: current_term: 1 leader_uuid: "2fbfe828d7454849ad7041b61b8a3866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fbfe828d7454849ad7041b61b8a3866" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43935 } } }
I20260929 16:26:54.117147 9696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:54.117169 9703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:26:54.120587 9703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:26:54.127238 9703 catalog_manager.cc:1347] Generated new cluster ID: 9bdbeea28b1c4bc6a64a9f49d26556bd
I20260929 16:26:54.127341 9703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:26:54.135645 9703 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:26:54.136940 9703 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:26:54.146629 9703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fbfe828d7454849ad7041b61b8a3866: Generated new TSK 0
I20260929 16:26:54.147528 9703 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:26:54.154112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43935
--builtin_ntp_servers=127.0.98.20:45389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 16:26:54.272320 9713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:54.272621 9713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:54.272697 9713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:54.277493 9713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:54.277577 9713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:54.277663 9713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:26:54.282641 9713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:54.283789 9713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:54.285295 9713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:54.295277 9718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:54.295233 9719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:54.295243 9721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:54.295507 9713 server_base.cc:1034] running on GCE node
I20260929 16:26:54.295903 9713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:54.296566 9713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:54.297736 9713 hybrid_clock.cc:648] HybridClock initialized: now 1790699214297711 us; error 38 us; skew 500 ppm
I20260929 16:26:54.299609 9713 webserver.cc:460] Webserver started at http://127.0.98.1:46125/ using document root <none> and password file <none>
I20260929 16:26:54.300133 9713 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:54.300194 9713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:54.300428 9713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:54.302574 9713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "92ca0b9b09a344c8a6c5b576979aa050"
format_stamp: "Formatted at 2026-09-29 16:26:54 on dist-test-slave-vq9v"
server_key: "bc7e65c74037623e5a7b9a16c19d4aa5"
server_key_iv: "12e59109b47bd772b2cc1d85b9027570"
server_key_version: "encryptionkey@0"
I20260929 16:26:54.303145 9713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "92ca0b9b09a344c8a6c5b576979aa050"
format_stamp: "Formatted at 2026-09-29 16:26:54 on dist-test-slave-vq9v"
server_key: "bc7e65c74037623e5a7b9a16c19d4aa5"
server_key_iv: "12e59109b47bd772b2cc1d85b9027570"
server_key_version: "encryptionkey@0"
I20260929 16:26:54.307129 9713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 16:26:54.309965 9727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:54.311440 9713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:54.311628 9713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "92ca0b9b09a344c8a6c5b576979aa050"
format_stamp: "Formatted at 2026-09-29 16:26:54 on dist-test-slave-vq9v"
server_key: "bc7e65c74037623e5a7b9a16c19d4aa5"
server_key_iv: "12e59109b47bd772b2cc1d85b9027570"
server_key_version: "encryptionkey@0"
I20260929 16:26:54.311766 9713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:54.337792 9713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:54.338598 9713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:54.338822 9713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:54.339491 9713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:54.340533 9713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:54.340584 9713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:54.340662 9713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:54.340695 9713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:54.350241 9713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46853
I20260929 16:26:54.350267 9840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46853 every 8 connection(s)
I20260929 16:26:54.351363 9713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:26:54.361485 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 9713
I20260929 16:26:54.361593 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:54.353445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35045 (local address 127.0.98.62:43935)
0929 16:26:54.353652 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:54.353656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:54.354791 (+ 1135us) server_negotiation.cc:407] Connection header received
0929 16:26:54.356350 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:54.356354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:54.356421 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:54.356521 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:54.357641 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:54.358260 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:54.359470 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:54.359699 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:54.359790 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:54.360920 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:54.360924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:54.360926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:54.360941 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:54.360982 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:54.361003 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:54.361071 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:54.361301 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 16:26:54.361384 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260929 16:26:54.361620 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:26:54.362007 392 external_mini_cluster.cc:1428] Setting key 96544fed6a1d48147051b03cebb7608f
I20260929 16:26:54.362525 9843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:54.353896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43935 (local address 127.0.98.1:35045)
0929 16:26:54.354625 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:54.354671 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:54.356100 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:54.356730 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:54.356737 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:54.356826 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:54.357438 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:54.357454 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:54.358382 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:54.358385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:54.359329 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:54.359338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:54.359841 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:54.359843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:54.359988 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:54.360636 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:26:54.360638 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:54.360679 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:26:54.360748 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:54.361089 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:54.361095 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:54.361100 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 16:26:54.361303 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 16:26:54.361607 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":120,"threads_started":1}
I20260929 16:26:54.363879 9841 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43935
I20260929 16:26:54.364148 9841 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:54.364250 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43935
--builtin_ntp_servers=127.0.98.20:45389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 16:26:54.364784 9841 heartbeater.cc:511] Master 127.0.98.62:43935 requested a full tablet report, sending...
I20260929 16:26:54.366405 9650 ts_manager.cc:194] Registered new tserver with Master: 92ca0b9b09a344c8a6c5b576979aa050 (127.0.98.1:46853)
I20260929 16:26:54.367576 9650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:35045
W20260929 16:26:54.478266 9844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:54.478576 9844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:54.478659 9844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:54.482851 9844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:54.482918 9844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:54.482975 9844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:26:54.487983 9844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:54.489257 9844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:54.490722 9844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:54.500984 9852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:54.501092 9849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:54.500989 9850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:54.501248 9844 server_base.cc:1034] running on GCE node
I20260929 16:26:54.501811 9844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:54.502516 9844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:54.503731 9844 hybrid_clock.cc:648] HybridClock initialized: now 1790699214503700 us; error 51 us; skew 500 ppm
I20260929 16:26:54.505934 9844 webserver.cc:460] Webserver started at http://127.0.98.2:42783/ using document root <none> and password file <none>
I20260929 16:26:54.506546 9844 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:54.506641 9844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:54.506879 9844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:54.508973 9844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "845c42d48e374a19888385478e271f59"
format_stamp: "Formatted at 2026-09-29 16:26:54 on dist-test-slave-vq9v"
server_key: "579316404278aa6ea3997476b28d74fa"
server_key_iv: "b281e6fed4c8f660aeacfe551f66c575"
server_key_version: "encryptionkey@0"
I20260929 16:26:54.509620 9844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "845c42d48e374a19888385478e271f59"
format_stamp: "Formatted at 2026-09-29 16:26:54 on dist-test-slave-vq9v"
server_key: "579316404278aa6ea3997476b28d74fa"
server_key_iv: "b281e6fed4c8f660aeacfe551f66c575"
server_key_version: "encryptionkey@0"
I20260929 16:26:54.513525 9844 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:26:54.516389 9858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:54.517982 9844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:54.518137 9844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "845c42d48e374a19888385478e271f59"
format_stamp: "Formatted at 2026-09-29 16:26:54 on dist-test-slave-vq9v"
server_key: "579316404278aa6ea3997476b28d74fa"
server_key_iv: "b281e6fed4c8f660aeacfe551f66c575"
server_key_version: "encryptionkey@0"
I20260929 16:26:54.518260 9844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:54.535734 9844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:54.536486 9844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:54.536696 9844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:54.537346 9844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:54.538293 9844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:54.538367 9844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:54.538439 9844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:54.538491 9844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:54.547693 9844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37927
I20260929 16:26:54.547712 9971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37927 every 8 connection(s)
I20260929 16:26:54.548614 9844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:26:54.552497 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 9844
I20260929 16:26:54.552603 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:26:54.552906 392 external_mini_cluster.cc:1428] Setting key 7db93c6a6852804489b35e5c98a75ed0
I20260929 16:26:54.555150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43935
--builtin_ntp_servers=127.0.98.20:45389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 16:26:54.559197 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:54.550486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41983 (local address 127.0.98.62:43935)
0929 16:26:54.550666 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:54.550669 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:54.551845 (+ 1176us) server_negotiation.cc:407] Connection header received
0929 16:26:54.553706 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:54.553710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:54.553767 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:54.553857 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:54.554974 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:54.555666 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:54.556954 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:54.557195 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:54.557389 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:54.558682 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:54.558686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:54.558688 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:54.558709 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:54.558728 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:54.558736 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:54.558814 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:54.558988 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 16:26:54.559055 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:26:54.560185 9974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:54.550872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43935 (local address 127.0.98.2:41983)
0929 16:26:54.551675 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:54.551719 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:54.553453 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:54.554047 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:54.554054 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:54.554149 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:54.554806 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:54.554823 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:54.555812 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:54.555823 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:54.556803 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:54.556813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:54.557433 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:54.557438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:54.557640 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:54.558347 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:26:54.558348 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:54.558392 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:26:54.558464 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:54.558828 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:54.558837 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:54.558843 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:26:54.558977 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 16:26:54.559299 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":125,"threads_started":1}
I20260929 16:26:54.561473 9972 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43935
I20260929 16:26:54.561728 9972 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:54.562275 9972 heartbeater.cc:511] Master 127.0.98.62:43935 requested a full tablet report, sending...
I20260929 16:26:54.563387 9650 ts_manager.cc:194] Registered new tserver with Master: 845c42d48e374a19888385478e271f59 (127.0.98.2:37927)
I20260929 16:26:54.563956 9650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41983
W20260929 16:26:54.676649 9975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:54.676939 9975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:54.677007 9975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:54.681028 9975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:54.681154 9975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:54.681269 9975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:26:54.686028 9975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:54.687108 9975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:54.688422 9975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:54.698151 9983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:54.698218 9981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:54.698383 9980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:54.698719 9975 server_base.cc:1034] running on GCE node
I20260929 16:26:54.699184 9975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:54.699862 9975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:54.701077 9975 hybrid_clock.cc:648] HybridClock initialized: now 1790699214701053 us; error 66 us; skew 500 ppm
I20260929 16:26:54.703231 9975 webserver.cc:460] Webserver started at http://127.0.98.3:41871/ using document root <none> and password file <none>
I20260929 16:26:54.703873 9975 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:54.703990 9975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:54.704253 9975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:54.706574 9975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "01f12747a4e14509930b37ad42cc6440"
format_stamp: "Formatted at 2026-09-29 16:26:54 on dist-test-slave-vq9v"
server_key: "d72e9fc5de8910a780e8d2dd8c7ac472"
server_key_iv: "85fb9b03434a740516e3d42332880cbf"
server_key_version: "encryptionkey@0"
I20260929 16:26:54.707217 9975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "01f12747a4e14509930b37ad42cc6440"
format_stamp: "Formatted at 2026-09-29 16:26:54 on dist-test-slave-vq9v"
server_key: "d72e9fc5de8910a780e8d2dd8c7ac472"
server_key_iv: "85fb9b03434a740516e3d42332880cbf"
server_key_version: "encryptionkey@0"
I20260929 16:26:54.712642 9975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 16:26:54.715829 9989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:54.717662 9975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:54.717782 9975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "01f12747a4e14509930b37ad42cc6440"
format_stamp: "Formatted at 2026-09-29 16:26:54 on dist-test-slave-vq9v"
server_key: "d72e9fc5de8910a780e8d2dd8c7ac472"
server_key_iv: "85fb9b03434a740516e3d42332880cbf"
server_key_version: "encryptionkey@0"
I20260929 16:26:54.717954 9975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:54.731341 9975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:54.732218 9975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:54.732474 9975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:54.733150 9975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:54.734664 9975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:54.734721 9975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:54.734766 9975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:54.734788 9975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:54.744547 9975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41531
I20260929 16:26:54.744575 10102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41531 every 8 connection(s)
I20260929 16:26:54.745713 9975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:26:54.752839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 9975
I20260929 16:26:54.752969 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:26:54.753347 392 external_mini_cluster.cc:1428] Setting key fd04b5eff4a33a8daac2f8f7a650ee58
I20260929 16:26:54.756726 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:54.747951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37055 (local address 127.0.98.62:43935)
0929 16:26:54.748134 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:54.748139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:54.749602 (+ 1463us) server_negotiation.cc:407] Connection header received
0929 16:26:54.751001 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:54.751004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:54.751059 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:54.751160 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:54.752187 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:54.752726 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:54.754538 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:54.754777 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:54.754930 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:54.756278 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:54.756281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:54.756283 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:54.756296 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:54.756317 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:54.756325 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:54.756387 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:54.756540 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 16:26:54.756602 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:26:54.757848 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:54.748455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43935 (local address 127.0.98.3:37055)
0929 16:26:54.749271 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:54.749334 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:54.750709 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:54.751352 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:54.751359 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:54.751459 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:54.752017 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:54.752032 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:54.752905 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:54.752910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:54.754382 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:54.754397 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:54.754992 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:54.754997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:54.755215 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:54.755961 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:26:54.755963 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:54.756017 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:26:54.756103 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:54.756402 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:54.756409 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:54.756413 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 16:26:54.756524 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 16:26:54.756880 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":145,"threads_started":1}
I20260929 16:26:54.759272 10103 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43935
I20260929 16:26:54.759568 10103 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:54.760218 10103 heartbeater.cc:511] Master 127.0.98.62:43935 requested a full tablet report, sending...
I20260929 16:26:54.761461 9650 ts_manager.cc:194] Registered new tserver with Master: 01f12747a4e14509930b37ad42cc6440 (127.0.98.3:41531)
I20260929 16:26:54.762102 9650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:37055
I20260929 16:26:54.768374 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 16:26:54.772025 9692 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:54.770905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43034 (local address 127.0.98.62:43935)
0929 16:26:54.771096 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:54.771100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:54.771154 (+ 54us) server_negotiation.cc:407] Connection header received
0929 16:26:54.771341 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:54.771344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:54.771392 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:54.771474 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:54.771879 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43034: BlockingRecv error: recv got EOF from 127.0.0.1:43034 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 16:26:54.772614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 9713
I20260929 16:26:54.778419 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 9844
I20260929 16:26:54.784466 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 9975
I20260929 16:26:54.790138 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 9622
2026-09-29T16:26:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (907 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T16:26:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:26:54Z Disabled control of system clock
I20260929 16:26:54.821720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45097
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 16:26:54.937981 10114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:54.938288 10114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:54.938329 10114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:54.942572 10114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:26:54.942654 10114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:54.942670 10114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:54.942687 10114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:26:54.942703 10114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:26:54.947911 10114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45097
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:54.949060 10114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:54.950819 10114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:54.960929 10122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:54.960939 10119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:54.960940 10114 server_base.cc:1034] running on GCE node
W20260929 16:26:54.960963 10120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:54.961903 10114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:54.963222 10114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:54.964442 10114 hybrid_clock.cc:648] HybridClock initialized: now 1790699214964429 us; error 35 us; skew 500 ppm
I20260929 16:26:54.966528 10114 webserver.cc:460] Webserver started at http://127.0.98.62:37549/ using document root <none> and password file <none>
I20260929 16:26:54.967125 10114 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:54.967218 10114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:54.967440 10114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:54.969586 10114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "df17612aeba34e41980809bfbe698fa4"
format_stamp: "Formatted at 2026-09-29 16:26:54 on dist-test-slave-vq9v"
server_key: "6bc7d5a1c015c5e6f517e68bb4b26221"
server_key_iv: "27dca2b734b982c990ced98ce91079ac"
server_key_version: "encryptionkey@0"
I20260929 16:26:54.970175 10114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "df17612aeba34e41980809bfbe698fa4"
format_stamp: "Formatted at 2026-09-29 16:26:54 on dist-test-slave-vq9v"
server_key: "6bc7d5a1c015c5e6f517e68bb4b26221"
server_key_iv: "27dca2b734b982c990ced98ce91079ac"
server_key_version: "encryptionkey@0"
I20260929 16:26:54.974079 10114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:26:54.976723 10128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:54.978242 10114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:26:54.978403 10114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "df17612aeba34e41980809bfbe698fa4"
format_stamp: "Formatted at 2026-09-29 16:26:54 on dist-test-slave-vq9v"
server_key: "6bc7d5a1c015c5e6f517e68bb4b26221"
server_key_iv: "27dca2b734b982c990ced98ce91079ac"
server_key_version: "encryptionkey@0"
I20260929 16:26:54.978543 10114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:54.983388 10114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:54.984052 10114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:54.984257 10114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:54.991613 10114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45097
I20260929 16:26:54.991624 10180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45097 every 8 connection(s)
I20260929 16:26:54.992751 10114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:26:54.996238 10181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:54.997990 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 10114
I20260929 16:26:54.998142 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:26:54.998449 392 external_mini_cluster.cc:1428] Setting key 41edff8bea3fefccdf3dcca19e98480b
I20260929 16:26:55.004191 10181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4: Bootstrap starting.
I20260929 16:26:55.005191 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:55.000299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42298 (local address 127.0.98.62:45097)
0929 16:26:55.001136 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:55.001154 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:55.001194 (+ 40us) server_negotiation.cc:407] Connection header received
0929 16:26:55.001991 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:55.002013 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:55.002405 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:55.002724 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:55.003696 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:55.003701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:55.003714 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:55.003725 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:55.003785 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:55.003803 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:55.003919 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:55.004069 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 16:26:55.004391 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":143,"threads_started":1}
I20260929 16:26:55.007035 10181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:55.007925 10181 log.cc:824] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:55.010121 10181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4: No bootstrap required, opened a new log
I20260929 16:26:55.014286 10181 raft_consensus.cc:348] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df17612aeba34e41980809bfbe698fa4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45097 } }
I20260929 16:26:55.014542 10181 raft_consensus.cc:374] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:55.014605 10181 raft_consensus.cc:729] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df17612aeba34e41980809bfbe698fa4, State: Initialized, Role: FOLLOWER
I20260929 16:26:55.015065 10181 consensus_queue.cc:260] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df17612aeba34e41980809bfbe698fa4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45097 } }
I20260929 16:26:55.015201 10181 raft_consensus.cc:388] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:26:55.015295 10181 raft_consensus.cc:482] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:26:55.015405 10181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:55.016436 10181 raft_consensus.cc:504] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df17612aeba34e41980809bfbe698fa4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45097 } }
I20260929 16:26:55.016817 10181 leader_election.cc:302] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df17612aeba34e41980809bfbe698fa4; no voters:
I20260929 16:26:55.017165 10181 leader_election.cc:288] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:26:55.017391 10186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:55.017696 10186 raft_consensus.cc:686] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4 [term 1 LEADER]: Becoming Leader. State: Replica: df17612aeba34e41980809bfbe698fa4, State: Running, Role: LEADER
I20260929 16:26:55.018025 10186 consensus_queue.cc:237] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df17612aeba34e41980809bfbe698fa4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45097 } }
I20260929 16:26:55.018577 10181 sys_catalog.cc:564] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:26:55.019732 10187 sys_catalog.cc:455] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df17612aeba34e41980809bfbe698fa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df17612aeba34e41980809bfbe698fa4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45097 } } }
I20260929 16:26:55.019872 10187 sys_catalog.cc:458] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:55.020182 10188 sys_catalog.cc:455] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df17612aeba34e41980809bfbe698fa4. Latest consensus state: current_term: 1 leader_uuid: "df17612aeba34e41980809bfbe698fa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df17612aeba34e41980809bfbe698fa4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45097 } } }
I20260929 16:26:55.020279 10188 sys_catalog.cc:458] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:55.020344 10195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:26:55.023609 10195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:26:55.029767 10195 catalog_manager.cc:1347] Generated new cluster ID: 1e7ef1e1f6fe4cd98e0e610e477d738d
I20260929 16:26:55.029846 10195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:26:55.072187 10195 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:26:55.072535 10195 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:26:55.081080 10195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df17612aeba34e41980809bfbe698fa4: Generated new TSK 0
I20260929 16:26:55.081893 10195 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:26:55.088552 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45097
--builtin_ntp_servers=127.0.98.20:42811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 16:26:55.214098 10205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:55.214428 10205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:55.214521 10205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:55.218899 10205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:55.219007 10205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:55.219113 10205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:26:55.224232 10205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:55.225533 10205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:55.226959 10205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:26:55.236948 10205 server_base.cc:1034] running on GCE node
W20260929 16:26:55.237030 10210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:55.237030 10211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:55.237059 10213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:55.237761 10205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:55.238459 10205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:55.239671 10205 hybrid_clock.cc:648] HybridClock initialized: now 1790699215239654 us; error 34 us; skew 500 ppm
I20260929 16:26:55.241807 10205 webserver.cc:460] Webserver started at http://127.0.98.1:38545/ using document root <none> and password file <none>
I20260929 16:26:55.242452 10205 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:55.242558 10205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:55.242803 10205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:55.244884 10205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "9d9ac1f41b8e40d0a3c93a53c7779c3e"
format_stamp: "Formatted at 2026-09-29 16:26:55 on dist-test-slave-vq9v"
server_key: "9fd101fd84fa33fe0ab1e773d97138af"
server_key_iv: "ea22f3c6e348f660f6ef526e5aa03df1"
server_key_version: "encryptionkey@0"
I20260929 16:26:55.245478 10205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9d9ac1f41b8e40d0a3c93a53c7779c3e"
format_stamp: "Formatted at 2026-09-29 16:26:55 on dist-test-slave-vq9v"
server_key: "9fd101fd84fa33fe0ab1e773d97138af"
server_key_iv: "ea22f3c6e348f660f6ef526e5aa03df1"
server_key_version: "encryptionkey@0"
I20260929 16:26:55.249388 10205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:26:55.252065 10219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:55.253497 10205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:55.253680 10205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "9d9ac1f41b8e40d0a3c93a53c7779c3e"
format_stamp: "Formatted at 2026-09-29 16:26:55 on dist-test-slave-vq9v"
server_key: "9fd101fd84fa33fe0ab1e773d97138af"
server_key_iv: "ea22f3c6e348f660f6ef526e5aa03df1"
server_key_version: "encryptionkey@0"
I20260929 16:26:55.253810 10205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:55.258531 10205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:55.259162 10205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:55.259354 10205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:55.259912 10205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:55.260830 10205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:55.260902 10205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:55.260970 10205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:55.261019 10205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:55.270316 10205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43369
I20260929 16:26:55.270354 10332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43369 every 8 connection(s)
I20260929 16:26:55.271250 10205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:26:55.276090 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 10205
I20260929 16:26:55.276212 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:26:55.276571 392 external_mini_cluster.cc:1428] Setting key b5fb2bd7aed019d4209bcd59f35b1285
I20260929 16:26:55.277771 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:55.273269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48585 (local address 127.0.98.62:45097)
0929 16:26:55.273430 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:55.273433 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:55.274512 (+ 1079us) server_negotiation.cc:407] Connection header received
0929 16:26:55.275793 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:55.275796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:55.275834 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:55.275900 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:55.277246 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:55.277253 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:55.277259 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:55.277275 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:55.277299 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:55.277316 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:55.277392 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:55.277609 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 16:26:55.277652 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:26:55.278775 10335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:55.273672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45097 (local address 127.0.98.1:48585)
0929 16:26:55.274337 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:55.274392 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:55.275581 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:55.276100 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:55.276111 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:55.276237 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:55.276897 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:26:55.276899 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:55.276954 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:26:55.277064 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:55.277400 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:55.277408 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:55.277414 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:26:55.277580 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 16:26:55.277780 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":107,"threads_started":1}
I20260929 16:26:55.278923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45097
--builtin_ntp_servers=127.0.98.20:42811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 16:26:55.280167 10333 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45097
I20260929 16:26:55.280436 10333 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:55.280952 10333 heartbeater.cc:511] Master 127.0.98.62:45097 requested a full tablet report, sending...
I20260929 16:26:55.282567 10145 ts_manager.cc:194] Registered new tserver with Master: 9d9ac1f41b8e40d0a3c93a53c7779c3e (127.0.98.1:43369)
W20260929 16:26:55.392879 10336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:55.393263 10336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:55.393355 10336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:55.397588 10336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:55.397656 10336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:55.397716 10336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:26:55.402639 10336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:55.403836 10336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:55.405326 10336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:55.415752 10342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:55.415921 10344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:55.415874 10341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:55.416219 10336 server_base.cc:1034] running on GCE node
I20260929 16:26:55.416666 10336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:55.417465 10336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:55.418665 10336 hybrid_clock.cc:648] HybridClock initialized: now 1790699215418643 us; error 37 us; skew 500 ppm
I20260929 16:26:55.420887 10336 webserver.cc:460] Webserver started at http://127.0.98.2:43947/ using document root <none> and password file <none>
I20260929 16:26:55.421588 10336 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:55.421691 10336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:55.421934 10336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:55.424089 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "a305affac1f44cf08b2fee9a03b156d6"
format_stamp: "Formatted at 2026-09-29 16:26:55 on dist-test-slave-vq9v"
server_key: "9e010869a9f537115f5eb0e0635e0502"
server_key_iv: "9de00d29f41a30671bf83638484f6e45"
server_key_version: "encryptionkey@0"
I20260929 16:26:55.424705 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a305affac1f44cf08b2fee9a03b156d6"
format_stamp: "Formatted at 2026-09-29 16:26:55 on dist-test-slave-vq9v"
server_key: "9e010869a9f537115f5eb0e0635e0502"
server_key_iv: "9de00d29f41a30671bf83638484f6e45"
server_key_version: "encryptionkey@0"
I20260929 16:26:55.428768 10336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:26:55.431679 10350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:55.433393 10336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:26:55.433553 10336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "a305affac1f44cf08b2fee9a03b156d6"
format_stamp: "Formatted at 2026-09-29 16:26:55 on dist-test-slave-vq9v"
server_key: "9e010869a9f537115f5eb0e0635e0502"
server_key_iv: "9de00d29f41a30671bf83638484f6e45"
server_key_version: "encryptionkey@0"
I20260929 16:26:55.433732 10336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:55.438668 10336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:55.439338 10336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:55.439545 10336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:55.440130 10336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:55.441107 10336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:55.441180 10336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:55.441296 10336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:55.441344 10336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:55.450338 10336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36663
I20260929 16:26:55.450361 10463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36663 every 8 connection(s)
I20260929 16:26:55.451282 10336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:26:55.456050 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 10336
I20260929 16:26:55.456185 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:26:55.456522 392 external_mini_cluster.cc:1428] Setting key b42b224383df1d3b75749aca49742f28
I20260929 16:26:55.457767 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:55.453235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33705 (local address 127.0.98.62:45097)
0929 16:26:55.453361 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:55.453365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:55.454483 (+ 1118us) server_negotiation.cc:407] Connection header received
0929 16:26:55.455821 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:55.455824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:55.455863 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:55.455940 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:55.457306 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:55.457309 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:55.457312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:55.457324 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:55.457347 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:55.457359 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:55.457454 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:55.457622 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 16:26:55.457660 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 16:26:55.458936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45097
--builtin_ntp_servers=127.0.98.20:42811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 16:26:55.459019 10466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:55.453664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45097 (local address 127.0.98.2:33705)
0929 16:26:55.454316 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:55.454358 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:55.455532 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:55.456137 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:55.456148 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:55.456275 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:55.457006 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:26:55.457007 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:55.457051 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:26:55.457122 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:55.457452 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:55.457461 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:55.457467 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:26:55.457667 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 16:26:55.457939 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":107,"threads_started":1}
I20260929 16:26:55.460224 10464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45097
I20260929 16:26:55.460481 10464 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:55.460901 10464 heartbeater.cc:511] Master 127.0.98.62:45097 requested a full tablet report, sending...
I20260929 16:26:55.462041 10145 ts_manager.cc:194] Registered new tserver with Master: a305affac1f44cf08b2fee9a03b156d6 (127.0.98.2:36663)
W20260929 16:26:55.572986 10467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:55.573382 10467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:55.573477 10467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:55.577564 10467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:55.577659 10467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:55.577759 10467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:26:55.582813 10467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:55.583973 10467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:55.585510 10467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:55.595786 10472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:55.595780 10475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:55.595842 10467 server_base.cc:1034] running on GCE node
W20260929 16:26:55.595744 10473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:55.596451 10467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:55.597313 10467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:55.598503 10467 hybrid_clock.cc:648] HybridClock initialized: now 1790699215598492 us; error 31 us; skew 500 ppm
I20260929 16:26:55.600600 10467 webserver.cc:460] Webserver started at http://127.0.98.3:43423/ using document root <none> and password file <none>
I20260929 16:26:55.601171 10467 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:55.601338 10467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:55.601542 10467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:55.603729 10467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "3572f8ea8a02418498dc206c88409ca6"
format_stamp: "Formatted at 2026-09-29 16:26:55 on dist-test-slave-vq9v"
server_key: "137bea50ada8250295d5e9d0deda57a9"
server_key_iv: "b64f77c763f3fb0e9c82613f8298791d"
server_key_version: "encryptionkey@0"
I20260929 16:26:55.604285 10467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3572f8ea8a02418498dc206c88409ca6"
format_stamp: "Formatted at 2026-09-29 16:26:55 on dist-test-slave-vq9v"
server_key: "137bea50ada8250295d5e9d0deda57a9"
server_key_iv: "b64f77c763f3fb0e9c82613f8298791d"
server_key_version: "encryptionkey@0"
I20260929 16:26:55.608359 10467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:26:55.611199 10481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:55.612720 10467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:26:55.612843 10467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "3572f8ea8a02418498dc206c88409ca6"
format_stamp: "Formatted at 2026-09-29 16:26:55 on dist-test-slave-vq9v"
server_key: "137bea50ada8250295d5e9d0deda57a9"
server_key_iv: "b64f77c763f3fb0e9c82613f8298791d"
server_key_version: "encryptionkey@0"
I20260929 16:26:55.612998 10467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:55.617905 10467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:55.618628 10467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:55.618784 10467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:55.619410 10467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:55.620383 10467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:55.620430 10467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:55.620469 10467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:55.620486 10467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:55.629451 10467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44445
I20260929 16:26:55.629478 10594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44445 every 8 connection(s)
I20260929 16:26:55.630390 10467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:26:55.635886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 10467
I20260929 16:26:55.636001 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:26:55.636323 392 external_mini_cluster.cc:1428] Setting key 3951c07a87820f28bfffc3faf4f07d83
I20260929 16:26:55.637634 10184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:55.632433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43955 (local address 127.0.98.62:45097)
0929 16:26:55.632601 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:55.632606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:55.633909 (+ 1303us) server_negotiation.cc:407] Connection header received
0929 16:26:55.635311 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:55.635315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:55.635363 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:55.635540 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:55.637039 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:55.637043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:55.637046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:26:55.637059 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:55.637087 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:26:55.637100 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:55.637290 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:55.637445 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 16:26:55.637496 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:26:55.639060 10597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:55.632926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45097 (local address 127.0.98.3:43955)
0929 16:26:55.633748 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:55.633795 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:55.635058 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:55.635742 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:55.635750 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:55.635848 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:55.636688 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:26:55.636689 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:55.636749 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:26:55.636881 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:55.637243 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:55.637254 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:55.637261 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 16:26:55.637554 (+ 293us) client_negotiation.cc:240] Negotiation successful
0929 16:26:55.637879 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":128,"threads_started":1}
I20260929 16:26:55.640259 10595 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45097
I20260929 16:26:55.640614 10595 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:55.641090 10595 heartbeater.cc:511] Master 127.0.98.62:45097 requested a full tablet report, sending...
I20260929 16:26:55.642336 10145 ts_manager.cc:194] Registered new tserver with Master: 3572f8ea8a02418498dc206c88409ca6 (127.0.98.3:44445)
I20260929 16:26:55.651333 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 16:26:55.654496 10184 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:26:55.653401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42310 (local address 127.0.98.62:45097)
0929 16:26:55.653563 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:55.653568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:55.653740 (+ 172us) server_negotiation.cc:407] Connection header received
0929 16:26:55.653926 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:55.653929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:55.653966 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:55.654077 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:55.654346 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42310: BlockingRecv error: recv got EOF from 127.0.0.1:42310 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 16:26:55.655035 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 10205
I20260929 16:26:55.660455 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 10336
I20260929 16:26:55.666078 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 10467
I20260929 16:26:55.671298 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 10114
2026-09-29T16:26:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (880 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:26:55 dist-test-slave-vq9v krb5kdc[10605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:26:55 dist-test-slave-vq9v krb5kdc[10605](info): set up 2 sockets
Sep 29 16:26:55 dist-test-slave-vq9v krb5kdc[10605](info): commencing operation
krb5kdc: starting...
W20260929 16:26:56.734970 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:26:56 dist-test-slave-vq9v krb5kdc[10605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:26:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:26:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:26:56.873457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33765
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:56.986275 10618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:56.986565 10618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:56.986608 10618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:56.990768 10618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:26:56.990840 10618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:56.990860 10618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:56.990877 10618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:26:56.990895 10618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:26:56.996358 10618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:56.997632 10618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:56.998726 10618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:57.007830 10626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:57.007956 10624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:57.007822 10623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:57.008193 10618 server_base.cc:1034] running on GCE node
I20260929 16:26:57.008862 10618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:57.010160 10618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:57.011386 10618 hybrid_clock.cc:648] HybridClock initialized: now 1790699217011369 us; error 43 us; skew 500 ppm
Sep 29 16:26:57 dist-test-slave-vq9v krb5kdc[10605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:57.022711 10618 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:26:57.023810 10618 webserver.cc:460] Webserver started at http://127.0.98.62:39427/ using document root <none> and password file <none>
I20260929 16:26:57.024372 10618 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:57.024435 10618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:57.024622 10618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:57.026870 10618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "5aa7c5fe227d406f9f88aa266504a96f"
format_stamp: "Formatted at 2026-09-29 16:26:57 on dist-test-slave-vq9v"
server_key: "97e24d145148a05ca0e3a074d07e762f"
server_key_iv: "b38dc28b646e377446b4e3c5944c4f9a"
server_key_version: "encryptionkey@0"
I20260929 16:26:57.027419 10618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "5aa7c5fe227d406f9f88aa266504a96f"
format_stamp: "Formatted at 2026-09-29 16:26:57 on dist-test-slave-vq9v"
server_key: "97e24d145148a05ca0e3a074d07e762f"
server_key_iv: "b38dc28b646e377446b4e3c5944c4f9a"
server_key_version: "encryptionkey@0"
I20260929 16:26:57.031816 10618 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:26:57.034508 10633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:57.035959 10618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:26:57.036084 10618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "5aa7c5fe227d406f9f88aa266504a96f"
format_stamp: "Formatted at 2026-09-29 16:26:57 on dist-test-slave-vq9v"
server_key: "97e24d145148a05ca0e3a074d07e762f"
server_key_iv: "b38dc28b646e377446b4e3c5944c4f9a"
server_key_version: "encryptionkey@0"
I20260929 16:26:57.036190 10618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:57.057086 10618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:57.060668 10618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:57.060874 10618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:57.068935 10618 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33765
I20260929 16:26:57.069368 10685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33765 every 8 connection(s)
I20260929 16:26:57.070115 10618 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:26:57.070659 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 10618
I20260929 16:26:57.070809 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:26:57.071122 392 external_mini_cluster.cc:1428] Setting key bdc8673e7b628a768ac98a5efa545c05
I20260929 16:26:57.073480 10686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:57.079193 10686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f: Bootstrap starting.
Sep 29 16:26:57 dist-test-slave-vq9v krb5kdc[10605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:57.081976 10686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:57.082763 10686 log.cc:824] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:57.085067 10686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f: No bootstrap required, opened a new log
I20260929 16:26:57.087028 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:57.072667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55614 (local address 127.0.98.62:33765)
0929 16:26:57.073083 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:57.073107 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:57.073143 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:26:57.073914 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:57.073933 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:57.074290 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:57.074646 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:57.075455 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.076447 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.077414 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.077896 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.078078 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:57.081025 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:57.081031 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:57.081046 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:57.081078 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:57.083687 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:57.084248 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:57.084261 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:57.084267 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:57.084394 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:57.084746 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:57.084754 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:57.084756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:57.085316 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:57.085534 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:57.085888 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 16:26:57.086228 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":140,"threads_started":1}
I20260929 16:26:57.088914 10686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa7c5fe227d406f9f88aa266504a96f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33765 } }
I20260929 16:26:57.089202 10686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:57.089304 10686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa7c5fe227d406f9f88aa266504a96f, State: Initialized, Role: FOLLOWER
I20260929 16:26:57.089751 10686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa7c5fe227d406f9f88aa266504a96f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33765 } }
I20260929 16:26:57.089919 10686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:26:57.090021 10686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:26:57.090154 10686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:57.091192 10686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa7c5fe227d406f9f88aa266504a96f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33765 } }
I20260929 16:26:57.091571 10686 leader_election.cc:302] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aa7c5fe227d406f9f88aa266504a96f; no voters:
I20260929 16:26:57.091974 10686 leader_election.cc:288] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:26:57.092049 10691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:57.092305 10691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f [term 1 LEADER]: Becoming Leader. State: Replica: 5aa7c5fe227d406f9f88aa266504a96f, State: Running, Role: LEADER
I20260929 16:26:57.092634 10691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa7c5fe227d406f9f88aa266504a96f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33765 } }
I20260929 16:26:57.093464 10686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:26:57.094524 10692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aa7c5fe227d406f9f88aa266504a96f. Latest consensus state: current_term: 1 leader_uuid: "5aa7c5fe227d406f9f88aa266504a96f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa7c5fe227d406f9f88aa266504a96f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33765 } } }
I20260929 16:26:57.094588 10693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5aa7c5fe227d406f9f88aa266504a96f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa7c5fe227d406f9f88aa266504a96f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33765 } } }
I20260929 16:26:57.094663 10692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:57.094694 10693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f [sys.catalog]: This master's current role is: LEADER
I20260929 16:26:57.096021 10700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:26:57.099011 10700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:26:57.105088 10700 catalog_manager.cc:1347] Generated new cluster ID: e82b27a35b8448be97b7acfa267ea74a
I20260929 16:26:57.105175 10700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:26:57.122577 10700 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:26:57.123967 10700 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:26:57.132913 10700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f: Generated new TSK 0
I20260929 16:26:57.133870 10700 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 16:26:57.173267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33765
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:57.315294 10712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:57.315601 10712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:57.315647 10712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:57.319947 10712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:57.320025 10712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:57.320089 10712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:26:57.325692 10712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:57.326905 10712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:57.328075 10712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:57.337944 10720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:57.338008 10717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:57.338213 10712 server_base.cc:1034] running on GCE node
W20260929 16:26:57.337953 10718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:57.338704 10712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:57.339355 10712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:57.340553 10712 hybrid_clock.cc:648] HybridClock initialized: now 1790699217340521 us; error 44 us; skew 500 ppm
Sep 29 16:26:57 dist-test-slave-vq9v krb5kdc[10605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:57.343629 10712 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:26:57.344612 10712 webserver.cc:460] Webserver started at http://127.0.98.1:37213/ using document root <none> and password file <none>
I20260929 16:26:57.345337 10712 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:57.345438 10712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:57.345669 10712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:57.347832 10712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "3261cc4dc4e4412db5a206497aa6f2e3"
format_stamp: "Formatted at 2026-09-29 16:26:57 on dist-test-slave-vq9v"
server_key: "2d1986f5e7a9808937d33983fbf202fc"
server_key_iv: "ee3ed69aec4f1a795417824e7bc81f98"
server_key_version: "encryptionkey@0"
I20260929 16:26:57.348446 10712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3261cc4dc4e4412db5a206497aa6f2e3"
format_stamp: "Formatted at 2026-09-29 16:26:57 on dist-test-slave-vq9v"
server_key: "2d1986f5e7a9808937d33983fbf202fc"
server_key_iv: "ee3ed69aec4f1a795417824e7bc81f98"
server_key_version: "encryptionkey@0"
I20260929 16:26:57.352596 10712 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:26:57.355362 10727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:57.356750 10712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:26:57.356901 10712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "3261cc4dc4e4412db5a206497aa6f2e3"
format_stamp: "Formatted at 2026-09-29 16:26:57 on dist-test-slave-vq9v"
server_key: "2d1986f5e7a9808937d33983fbf202fc"
server_key_iv: "ee3ed69aec4f1a795417824e7bc81f98"
server_key_version: "encryptionkey@0"
I20260929 16:26:57.357030 10712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:57.374714 10712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:57.377942 10712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:57.378156 10712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:57.378756 10712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:57.379588 10712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:57.379675 10712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:57.379748 10712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:57.379802 10712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:57.389139 10712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35531
I20260929 16:26:57.389179 10840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35531 every 8 connection(s)
I20260929 16:26:57.390146 10712 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:26:57.390928 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 10712
I20260929 16:26:57.391080 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:26:57.391371 392 external_mini_cluster.cc:1428] Setting key 0733acdfcd83aaa31df913a9d1d828d6
Sep 29 16:26:57 dist-test-slave-vq9v krb5kdc[10605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 16:26:57.407383 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:57.392339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51457 (local address 127.0.98.62:33765)
0929 16:26:57.392509 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:57.392515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:57.393821 (+ 1306us) server_negotiation.cc:407] Connection header received
0929 16:26:57.395686 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:57.395690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:57.395747 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:57.395837 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:57.397163 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.397776 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.399056 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.399558 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.399710 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:57.402975 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:57.402979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:57.402981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:57.403009 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:57.404766 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:57.405545 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:57.405549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:57.405551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:57.405699 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:57.406168 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:57.406171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:57.406173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:57.406396 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:57.406506 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:57.407067 (+ 561us) server_negotiation.cc:299] Negotiation successful
0929 16:26:57.407214 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:26:57.408277 10843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:57.392959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33765 (local address 127.0.98.1:51457)
0929 16:26:57.393634 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:57.393701 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:57.395446 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:57.396060 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:57.396087 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:57.396470 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:57.396982 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:57.397010 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.397955 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.397959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:57.398899 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:57.398909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.399765 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.399769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:57.399950 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:57.400714 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:57.400732 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:57.402726 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:57.404927 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:57.404934 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:57.404959 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:57.405424 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:57.405801 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:57.405807 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:57.405812 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:57.406030 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:57.406519 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:57.406527 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:57.406806 (+ 279us) client_negotiation.cc:769] Sending connection context
0929 16:26:57.407004 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 16:26:57.407375 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":213,"threads_started":1}
I20260929 16:26:57.409840 10841 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33765
I20260929 16:26:57.410128 10841 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:57.410684 10841 heartbeater.cc:511] Master 127.0.98.62:33765 requested a full tablet report, sending...
I20260929 16:26:57.412499 10650 ts_manager.cc:194] Registered new tserver with Master: 3261cc4dc4e4412db5a206497aa6f2e3 (127.0.98.1:35531)
I20260929 16:26:57.414333 10650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 16:26:57.425635 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33765
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:57.540175 10846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:57.540443 10846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:57.540539 10846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:57.544682 10846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:57.544747 10846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:57.544806 10846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:26:57.550168 10846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:57.551167 10846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:57.552136 10846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:26:57.561993 10851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:57.562116 10846 server_base.cc:1034] running on GCE node
W20260929 16:26:57.562043 10852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:57.562067 10854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:57.562822 10846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:57.563467 10846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:57.564670 10846 hybrid_clock.cc:648] HybridClock initialized: now 1790699217564633 us; error 51 us; skew 500 ppm
Sep 29 16:26:57 dist-test-slave-vq9v krb5kdc[10605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:57.567483 10846 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:26:57.568459 10846 webserver.cc:460] Webserver started at http://127.0.98.2:33995/ using document root <none> and password file <none>
I20260929 16:26:57.569029 10846 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:57.569121 10846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:57.569381 10846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:57.571416 10846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "8d725d18bcc842bd8a101f6fb8ea27be"
format_stamp: "Formatted at 2026-09-29 16:26:57 on dist-test-slave-vq9v"
server_key: "81c23e69feaeaec62973f6368424f585"
server_key_iv: "40fe0a81732805eca9e17eb4105bbca3"
server_key_version: "encryptionkey@0"
I20260929 16:26:57.572003 10846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8d725d18bcc842bd8a101f6fb8ea27be"
format_stamp: "Formatted at 2026-09-29 16:26:57 on dist-test-slave-vq9v"
server_key: "81c23e69feaeaec62973f6368424f585"
server_key_iv: "40fe0a81732805eca9e17eb4105bbca3"
server_key_version: "encryptionkey@0"
I20260929 16:26:57.575959 10846 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:26:57.578600 10861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:57.580163 10846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:26:57.580322 10846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "8d725d18bcc842bd8a101f6fb8ea27be"
format_stamp: "Formatted at 2026-09-29 16:26:57 on dist-test-slave-vq9v"
server_key: "81c23e69feaeaec62973f6368424f585"
server_key_iv: "40fe0a81732805eca9e17eb4105bbca3"
server_key_version: "encryptionkey@0"
I20260929 16:26:57.580451 10846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:57.597402 10846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:57.600988 10846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:57.601253 10846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:57.601903 10846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:57.602805 10846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:57.602851 10846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:57.602891 10846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:57.602906 10846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:57.612628 10846 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41547
I20260929 16:26:57.612660 10974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41547 every 8 connection(s)
I20260929 16:26:57.613646 10846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:26:57.622519 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 10846
I20260929 16:26:57.622653 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:26:57.622921 392 external_mini_cluster.cc:1428] Setting key abe81443d48484ec0359dc1cae0edfaf
Sep 29 16:26:57 dist-test-slave-vq9v krb5kdc[10605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 16:26:57.634560 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:57.615625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45321 (local address 127.0.98.62:33765)
0929 16:26:57.615786 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:57.615789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:57.616955 (+ 1166us) server_negotiation.cc:407] Connection header received
0929 16:26:57.617940 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:57.617943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:57.618007 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:57.618102 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:57.619407 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.619988 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.621363 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.621647 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.621768 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:57.630498 (+ 8730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:57.630502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:57.630504 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:57.630531 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:57.632057 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:57.632639 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:57.632642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:57.632643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:57.632773 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:57.633253 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:57.633256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:57.633258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:57.633451 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:57.633530 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:57.634147 (+ 617us) server_negotiation.cc:299] Negotiation successful
0929 16:26:57.634383 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:26:57.635535 10977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:57.616036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33765 (local address 127.0.98.2:45321)
0929 16:26:57.616789 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:57.616834 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:57.617693 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:57.618277 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:57.618284 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:57.618736 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:57.619234 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:57.619248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.620141 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.620144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:57.621151 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:57.621161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.621827 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.621830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:57.621980 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:57.628340 (+ 6360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:57.628370 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:57.630273 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:57.632186 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:57.632193 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:57.632222 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:57.632492 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:57.632896 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:57.632899 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:57.632900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:57.633103 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:57.633588 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:57.633593 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:57.633868 (+ 275us) client_negotiation.cc:769] Sending connection context
0929 16:26:57.634042 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 16:26:57.634395 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":115,"threads_started":1}
I20260929 16:26:57.637908 10975 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33765
I20260929 16:26:57.638181 10975 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:57.638841 10975 heartbeater.cc:511] Master 127.0.98.62:33765 requested a full tablet report, sending...
I20260929 16:26:57.640061 10650 ts_manager.cc:194] Registered new tserver with Master: 8d725d18bcc842bd8a101f6fb8ea27be (127.0.98.2:41547)
I20260929 16:26:57.640734 10650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 16:26:57.652294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33765
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:26:57.771073 10980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:26:57.771353 10980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:26:57.771399 10980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:26:57.775537 10980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:26:57.775600 10980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:26:57.775708 10980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:26:57.781131 10980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:26:57.782223 10980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:26:57.783186 10980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:26:57.793855 10980 server_base.cc:1034] running on GCE node
W20260929 16:26:57.793807 10986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:57.793885 10988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:26:57.793807 10985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:26:57.794492 10980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:26:57.795279 10980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:26:57.796476 10980 hybrid_clock.cc:648] HybridClock initialized: now 1790699217796460 us; error 38 us; skew 500 ppm
Sep 29 16:26:57 dist-test-slave-vq9v krb5kdc[10605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:26:57.799531 10980 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:26:57.800525 10980 webserver.cc:460] Webserver started at http://127.0.98.3:39799/ using document root <none> and password file <none>
I20260929 16:26:57.801136 10980 fs_manager.cc:359] Metadata directory not provided
I20260929 16:26:57.801260 10980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:26:57.801508 10980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:26:57.803673 10980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "ec2b1502ebc44bd2ac016cb35d72e6fa"
format_stamp: "Formatted at 2026-09-29 16:26:57 on dist-test-slave-vq9v"
server_key: "100c72c3561205280eef0f4cc0beaf18"
server_key_iv: "509a1237fabf74fce82607fba3aa2399"
server_key_version: "encryptionkey@0"
I20260929 16:26:57.804272 10980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ec2b1502ebc44bd2ac016cb35d72e6fa"
format_stamp: "Formatted at 2026-09-29 16:26:57 on dist-test-slave-vq9v"
server_key: "100c72c3561205280eef0f4cc0beaf18"
server_key_iv: "509a1237fabf74fce82607fba3aa2399"
server_key_version: "encryptionkey@0"
I20260929 16:26:57.808313 10980 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 16:26:57.811053 10995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:57.812562 10980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:26:57.812728 10980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "ec2b1502ebc44bd2ac016cb35d72e6fa"
format_stamp: "Formatted at 2026-09-29 16:26:57 on dist-test-slave-vq9v"
server_key: "100c72c3561205280eef0f4cc0beaf18"
server_key_iv: "509a1237fabf74fce82607fba3aa2399"
server_key_version: "encryptionkey@0"
I20260929 16:26:57.812860 10980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:26:57.830119 10980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:26:57.833771 10980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:26:57.834002 10980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:26:57.834685 10980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:26:57.835598 10980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:26:57.835671 10980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:57.835742 10980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:26:57.835793 10980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:26:57.845453 10980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44023
I20260929 16:26:57.845480 11108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44023 every 8 connection(s)
I20260929 16:26:57.846374 10980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:26:57.849476 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 10980
I20260929 16:26:57.849658 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:26:57.849995 392 external_mini_cluster.cc:1428] Setting key 3a2658e97c382f0224c52566ea948532
Sep 29 16:26:57 dist-test-slave-vq9v krb5kdc[10605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:26:57.861703 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:57.848257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35731 (local address 127.0.98.62:33765)
0929 16:26:57.848428 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:57.848433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:57.849662 (+ 1229us) server_negotiation.cc:407] Connection header received
0929 16:26:57.850811 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:57.850813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:57.850904 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:57.851002 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:57.852597 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.853144 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.854389 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.854625 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.854792 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:57.857471 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:57.857474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:57.857477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:57.857609 (+ 132us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:57.859087 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:57.859670 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:57.859674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:57.859676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:57.859780 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:57.860251 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:57.860254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:57.860255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:57.860577 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:57.860709 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:57.861371 (+ 662us) server_negotiation.cc:299] Negotiation successful
0929 16:26:57.861494 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:26:57.862480 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:57.848682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33765 (local address 127.0.98.3:35731)
0929 16:26:57.849482 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:57.849543 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:57.850561 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:57.851204 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:57.851211 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:57.851736 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:26:57.852427 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:57.852447 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.853335 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.853337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:57.854262 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:57.854271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.854855 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.854859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:57.855033 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:57.855677 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:26:57.855695 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:26:57.857283 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:26:57.859215 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:57.859220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:57.859234 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:57.859519 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:57.859970 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:26:57.859972 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:26:57.859974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:26:57.860144 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:26:57.860715 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:26:57.860720 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:26:57.861054 (+ 334us) client_negotiation.cc:769] Sending connection context
0929 16:26:57.861291 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 16:26:57.861570 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":112,"threads_started":1}
I20260929 16:26:57.863873 11109 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33765
I20260929 16:26:57.864125 11109 heartbeater.cc:464] Registering TS with master...
I20260929 16:26:57.864593 11109 heartbeater.cc:511] Master 127.0.98.62:33765 requested a full tablet report, sending...
I20260929 16:26:57.865618 10650 ts_manager.cc:194] Registered new tserver with Master: ec2b1502ebc44bd2ac016cb35d72e6fa (127.0.98.3:44023)
I20260929 16:26:57.866225 10650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35731
I20260929 16:26:57.877991 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:26:57.887701 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:57.879943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55630 (local address 127.0.98.62:33765)
0929 16:26:57.880151 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:57.880155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:57.880227 (+ 72us) server_negotiation.cc:407] Connection header received
0929 16:26:57.880333 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:57.880335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:57.880376 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:57.880443 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:26:57.881162 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.881746 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.882601 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.882813 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.882891 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:57.884457 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:26:57.884459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:26:57.884461 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:26:57.884486 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:26:57.886068 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:57.886457 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:57.886460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:57.886461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:57.886531 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:26:57.886767 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:26:57.886769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:26:57.886770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:26:57.886974 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:26:57.887079 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:57.887370 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 16:26:57.887512 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260929 16:26:57.891113 10650 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:26:57.894426 10650 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:26:57.908900 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:57.903429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42406 (local address 127.0.98.1:35531)
0929 16:26:57.903838 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:57.903855 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:57.903885 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:26:57.904141 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:57.904147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:57.904383 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:57.904543 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:57.905040 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.906195 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.907570 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.908220 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.908351 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:57.908519 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:57.908652 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 16:26:57.908747 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":106,"threads_started":1}
I20260929 16:26:57.908891 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:57.903277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35531 (local address 127.0.0.1:42406)
0929 16:26:57.903730 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:57.903778 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:57.903912 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:57.904759 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:57.904765 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:57.904798 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:57.904932 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:57.904938 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.906317 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.906323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:57.907435 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:57.907444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.908350 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.908353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:57.908461 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:57.908496 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:26:57.908615 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 16:26:57.908739 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":43,"thread_start_us":87,"threads_started":1}
I20260929 16:26:57.910594 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:57.903426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44023 (local address 127.0.0.1:47134)
0929 16:26:57.904585 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:57.904600 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:57.904677 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:57.905444 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:57.905447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:57.905462 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:57.905545 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:57.905551 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.907102 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.907106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:57.908851 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:57.908859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.910171 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.910176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:57.910267 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:57.910292 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:26:57.910367 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 16:26:57.910441 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"mutex_wait_us":92,"thread_start_us":169,"threads_started":1}
I20260929 16:26:57.910857 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:57.904027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41547 (local address 127.0.0.1:37218)
0929 16:26:57.905152 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:57.905165 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:57.905285 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:57.905748 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:57.905751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:57.905767 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:57.905849 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:57.905855 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.907280 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.907283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:57.908669 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:57.908680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.910130 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.910134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:57.910237 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:57.910263 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 16:26:57.910346 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 16:26:57.910427 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"mutex_wait_us":44,"thread_start_us":97,"threads_started":1}
I20260929 16:26:57.911021 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:57.904450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47134 (local address 127.0.98.3:44023)
0929 16:26:57.904882 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:57.904898 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:57.904924 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:26:57.905004 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:57.905010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:57.905304 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:57.905473 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:57.905626 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.906914 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.909145 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.910020 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.910147 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:57.910280 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:57.910795 (+ 515us) server_negotiation.cc:299] Negotiation successful
0929 16:26:57.910894 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":129,"threads_started":1}
I20260929 16:26:57.911809 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:57.904228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37218 (local address 127.0.98.2:41547)
0929 16:26:57.905068 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:57.905081 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:57.905267 (+ 186us) server_negotiation.cc:407] Connection header received
0929 16:26:57.905394 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:57.905403 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:57.905602 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:57.905765 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:57.905947 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.907135 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.908837 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:57.909791 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:57.909906 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:57.910058 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:57.910379 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 16:26:57.910476 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":100,"threads_started":1}
I20260929 16:26:57.912240 10775 tablet_service.cc:1467] Processing CreateTablet for tablet 52f463984566462bb17eb2843ced39b5 (DEFAULT_TABLE table=test-table [id=0a36969e91be4ccbb963d0d611a3fcbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:57.913508 10775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f463984566462bb17eb2843ced39b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:57.913965 11043 tablet_service.cc:1467] Processing CreateTablet for tablet 52f463984566462bb17eb2843ced39b5 (DEFAULT_TABLE table=test-table [id=0a36969e91be4ccbb963d0d611a3fcbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:57.914386 10909 tablet_service.cc:1467] Processing CreateTablet for tablet 52f463984566462bb17eb2843ced39b5 (DEFAULT_TABLE table=test-table [id=0a36969e91be4ccbb963d0d611a3fcbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:26:57.915056 11043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f463984566462bb17eb2843ced39b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:57.915313 10909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f463984566462bb17eb2843ced39b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:26:57.921185 11126 tablet_bootstrap.cc:492] T 52f463984566462bb17eb2843ced39b5 P 3261cc4dc4e4412db5a206497aa6f2e3: Bootstrap starting.
I20260929 16:26:57.922489 11128 tablet_bootstrap.cc:492] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa: Bootstrap starting.
I20260929 16:26:57.922680 11127 tablet_bootstrap.cc:492] T 52f463984566462bb17eb2843ced39b5 P 8d725d18bcc842bd8a101f6fb8ea27be: Bootstrap starting.
I20260929 16:26:57.924561 11128 tablet_bootstrap.cc:654] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:57.924741 11126 tablet_bootstrap.cc:654] T 52f463984566462bb17eb2843ced39b5 P 3261cc4dc4e4412db5a206497aa6f2e3: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:57.925173 11127 tablet_bootstrap.cc:654] T 52f463984566462bb17eb2843ced39b5 P 8d725d18bcc842bd8a101f6fb8ea27be: Neither blocks nor log segments found. Creating new log.
I20260929 16:26:57.925724 11128 log.cc:824] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:57.925732 11126 log.cc:824] T 52f463984566462bb17eb2843ced39b5 P 3261cc4dc4e4412db5a206497aa6f2e3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:57.925961 11127 log.cc:824] T 52f463984566462bb17eb2843ced39b5 P 8d725d18bcc842bd8a101f6fb8ea27be: Log is configured to *not* fsync() on all Append() calls
I20260929 16:26:57.927626 11128 tablet_bootstrap.cc:492] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa: No bootstrap required, opened a new log
I20260929 16:26:57.927744 11126 tablet_bootstrap.cc:492] T 52f463984566462bb17eb2843ced39b5 P 3261cc4dc4e4412db5a206497aa6f2e3: No bootstrap required, opened a new log
I20260929 16:26:57.927744 11127 tablet_bootstrap.cc:492] T 52f463984566462bb17eb2843ced39b5 P 8d725d18bcc842bd8a101f6fb8ea27be: No bootstrap required, opened a new log
I20260929 16:26:57.927862 11128 ts_tablet_manager.cc:1397] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 16:26:57.927925 11127 ts_tablet_manager.cc:1397] T 52f463984566462bb17eb2843ced39b5 P 8d725d18bcc842bd8a101f6fb8ea27be: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:26:57.927927 11126 ts_tablet_manager.cc:1397] T 52f463984566462bb17eb2843ced39b5 P 3261cc4dc4e4412db5a206497aa6f2e3: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 16:26:57.931167 11128 raft_consensus.cc:348] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3261cc4dc4e4412db5a206497aa6f2e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35531 } } peers { permanent_uuid: "ec2b1502ebc44bd2ac016cb35d72e6fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "8d725d18bcc842bd8a101f6fb8ea27be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41547 } }
I20260929 16:26:57.931449 11128 raft_consensus.cc:374] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:57.931504 11128 raft_consensus.cc:729] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec2b1502ebc44bd2ac016cb35d72e6fa, State: Initialized, Role: FOLLOWER
I20260929 16:26:57.931973 11128 consensus_queue.cc:260] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3261cc4dc4e4412db5a206497aa6f2e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35531 } } peers { permanent_uuid: "ec2b1502ebc44bd2ac016cb35d72e6fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "8d725d18bcc842bd8a101f6fb8ea27be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41547 } }
I20260929 16:26:57.931922 11127 raft_consensus.cc:348] T 52f463984566462bb17eb2843ced39b5 P 8d725d18bcc842bd8a101f6fb8ea27be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3261cc4dc4e4412db5a206497aa6f2e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35531 } } peers { permanent_uuid: "ec2b1502ebc44bd2ac016cb35d72e6fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "8d725d18bcc842bd8a101f6fb8ea27be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41547 } }
I20260929 16:26:57.932237 11127 raft_consensus.cc:374] T 52f463984566462bb17eb2843ced39b5 P 8d725d18bcc842bd8a101f6fb8ea27be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:57.932300 11127 raft_consensus.cc:729] T 52f463984566462bb17eb2843ced39b5 P 8d725d18bcc842bd8a101f6fb8ea27be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d725d18bcc842bd8a101f6fb8ea27be, State: Initialized, Role: FOLLOWER
I20260929 16:26:57.932304 11126 raft_consensus.cc:348] T 52f463984566462bb17eb2843ced39b5 P 3261cc4dc4e4412db5a206497aa6f2e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3261cc4dc4e4412db5a206497aa6f2e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35531 } } peers { permanent_uuid: "ec2b1502ebc44bd2ac016cb35d72e6fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "8d725d18bcc842bd8a101f6fb8ea27be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41547 } }
I20260929 16:26:57.932555 11126 raft_consensus.cc:374] T 52f463984566462bb17eb2843ced39b5 P 3261cc4dc4e4412db5a206497aa6f2e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:26:57.932608 11126 raft_consensus.cc:729] T 52f463984566462bb17eb2843ced39b5 P 3261cc4dc4e4412db5a206497aa6f2e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3261cc4dc4e4412db5a206497aa6f2e3, State: Initialized, Role: FOLLOWER
I20260929 16:26:57.932754 11127 consensus_queue.cc:260] T 52f463984566462bb17eb2843ced39b5 P 8d725d18bcc842bd8a101f6fb8ea27be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3261cc4dc4e4412db5a206497aa6f2e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35531 } } peers { permanent_uuid: "ec2b1502ebc44bd2ac016cb35d72e6fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "8d725d18bcc842bd8a101f6fb8ea27be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41547 } }
I20260929 16:26:57.932974 11126 consensus_queue.cc:260] T 52f463984566462bb17eb2843ced39b5 P 3261cc4dc4e4412db5a206497aa6f2e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3261cc4dc4e4412db5a206497aa6f2e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35531 } } peers { permanent_uuid: "ec2b1502ebc44bd2ac016cb35d72e6fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "8d725d18bcc842bd8a101f6fb8ea27be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41547 } }
I20260929 16:26:57.933224 11128 ts_tablet_manager.cc:1428] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa: Time spent starting tablet: real 0.005s user 0.001s sys 0.006s
I20260929 16:26:57.933564 10975 heartbeater.cc:503] Master 127.0.98.62:33765 was elected leader, sending a full tablet report...
I20260929 16:26:57.933732 11127 ts_tablet_manager.cc:1428] T 52f463984566462bb17eb2843ced39b5 P 8d725d18bcc842bd8a101f6fb8ea27be: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:26:57.933811 11126 ts_tablet_manager.cc:1428] T 52f463984566462bb17eb2843ced39b5 P 3261cc4dc4e4412db5a206497aa6f2e3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:26:57.934140 10841 heartbeater.cc:503] Master 127.0.98.62:33765 was elected leader, sending a full tablet report...
I20260929 16:26:57.934460 11109 heartbeater.cc:503] Master 127.0.98.62:33765 was elected leader, sending a full tablet report...
I20260929 16:26:58.038030 11132 raft_consensus.cc:482] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:26:58.038321 11132 raft_consensus.cc:504] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3261cc4dc4e4412db5a206497aa6f2e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35531 } } peers { permanent_uuid: "ec2b1502ebc44bd2ac016cb35d72e6fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "8d725d18bcc842bd8a101f6fb8ea27be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41547 } }
I20260929 16:26:58.039784 11132 leader_election.cc:288] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3261cc4dc4e4412db5a206497aa6f2e3 (127.0.98.1:35531), 8d725d18bcc842bd8a101f6fb8ea27be (127.0.98.2:41547)
I20260929 16:26:58.044051 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:58.040358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53815 (local address 127.0.98.2:41547)
0929 16:26:58.040527 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:58.040533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:58.040762 (+ 229us) server_negotiation.cc:407] Connection header received
0929 16:26:58.040821 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:58.040823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:58.040871 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:58.040962 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:58.041317 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:58.041793 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:58.042991 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:58.043400 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:58.043483 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:58.043578 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:58.043839 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 16:26:58.043907 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:26:58.044075 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:58.040070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35531 (local address 127.0.98.3:37181)
0929 16:26:58.040244 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:58.040265 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:58.040383 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:58.040789 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:58.040792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:58.040814 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:58.040945 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:58.040955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:58.041911 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:58.041916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:58.042808 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:58.042817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:58.043578 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:58.043581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:58.043719 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:58.043735 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 16:26:58.043829 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 16:26:58.043903 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":50}
I20260929 16:26:58.044085 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:58.040224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37181 (local address 127.0.98.1:35531)
0929 16:26:58.040369 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:58.040374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:58.040393 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:26:58.040555 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:58.040559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:58.040619 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:58.040720 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:26:58.041085 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:58.041782 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:58.042953 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:58.043418 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:58.043558 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:58.043624 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:58.043868 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 16:26:58.043944 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:26:58.044094 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:58.040156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41547 (local address 127.0.98.3:53815)
0929 16:26:58.040564 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:58.040632 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 16:26:58.040738 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:26:58.040986 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:26:58.040988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:26:58.041029 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:26:58.041157 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:58.041164 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:58.041918 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:58.041924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:58.042818 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:26:58.042827 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:26:58.043540 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:26:58.043544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:26:58.043689 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:58.043724 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:26:58.043790 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 16:26:58.043899 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":5,"thread_start_us":144,"threads_started":1}
I20260929 16:26:58.044821 10795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f463984566462bb17eb2843ced39b5" candidate_uuid: "ec2b1502ebc44bd2ac016cb35d72e6fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3261cc4dc4e4412db5a206497aa6f2e3" is_pre_election: true
I20260929 16:26:58.045017 10929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f463984566462bb17eb2843ced39b5" candidate_uuid: "ec2b1502ebc44bd2ac016cb35d72e6fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d725d18bcc842bd8a101f6fb8ea27be" is_pre_election: true
I20260929 16:26:58.045182 10795 raft_consensus.cc:2454] T 52f463984566462bb17eb2843ced39b5 P 3261cc4dc4e4412db5a206497aa6f2e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec2b1502ebc44bd2ac016cb35d72e6fa in term 0.
I20260929 16:26:58.045303 10929 raft_consensus.cc:2454] T 52f463984566462bb17eb2843ced39b5 P 8d725d18bcc842bd8a101f6fb8ea27be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec2b1502ebc44bd2ac016cb35d72e6fa in term 0.
I20260929 16:26:58.045773 10996 leader_election.cc:302] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3261cc4dc4e4412db5a206497aa6f2e3, 8d725d18bcc842bd8a101f6fb8ea27be, ec2b1502ebc44bd2ac016cb35d72e6fa; no voters:
I20260929 16:26:58.046064 11132 raft_consensus.cc:2781] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:26:58.046152 11132 raft_consensus.cc:482] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:26:58.046195 11132 raft_consensus.cc:3037] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:58.047209 11132 raft_consensus.cc:504] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3261cc4dc4e4412db5a206497aa6f2e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35531 } } peers { permanent_uuid: "ec2b1502ebc44bd2ac016cb35d72e6fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "8d725d18bcc842bd8a101f6fb8ea27be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41547 } }
I20260929 16:26:58.047549 11132 leader_election.cc:288] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa [CANDIDATE]: Term 1 election: Requested vote from peers 3261cc4dc4e4412db5a206497aa6f2e3 (127.0.98.1:35531), 8d725d18bcc842bd8a101f6fb8ea27be (127.0.98.2:41547)
I20260929 16:26:58.047940 10795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f463984566462bb17eb2843ced39b5" candidate_uuid: "ec2b1502ebc44bd2ac016cb35d72e6fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3261cc4dc4e4412db5a206497aa6f2e3"
I20260929 16:26:58.047945 10929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f463984566462bb17eb2843ced39b5" candidate_uuid: "ec2b1502ebc44bd2ac016cb35d72e6fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d725d18bcc842bd8a101f6fb8ea27be"
I20260929 16:26:58.048105 10929 raft_consensus.cc:3037] T 52f463984566462bb17eb2843ced39b5 P 8d725d18bcc842bd8a101f6fb8ea27be [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:58.048051 10795 raft_consensus.cc:3037] T 52f463984566462bb17eb2843ced39b5 P 3261cc4dc4e4412db5a206497aa6f2e3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:26:58.049155 10795 raft_consensus.cc:2454] T 52f463984566462bb17eb2843ced39b5 P 3261cc4dc4e4412db5a206497aa6f2e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec2b1502ebc44bd2ac016cb35d72e6fa in term 1.
I20260929 16:26:58.049324 10929 raft_consensus.cc:2454] T 52f463984566462bb17eb2843ced39b5 P 8d725d18bcc842bd8a101f6fb8ea27be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec2b1502ebc44bd2ac016cb35d72e6fa in term 1.
I20260929 16:26:58.049577 10998 leader_election.cc:302] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3261cc4dc4e4412db5a206497aa6f2e3, ec2b1502ebc44bd2ac016cb35d72e6fa; no voters:
I20260929 16:26:58.049750 11132 raft_consensus.cc:2781] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:26:58.050004 11132 raft_consensus.cc:686] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa [term 1 LEADER]: Becoming Leader. State: Replica: ec2b1502ebc44bd2ac016cb35d72e6fa, State: Running, Role: LEADER
I20260929 16:26:58.050400 11132 consensus_queue.cc:237] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3261cc4dc4e4412db5a206497aa6f2e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35531 } } peers { permanent_uuid: "ec2b1502ebc44bd2ac016cb35d72e6fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } } peers { permanent_uuid: "8d725d18bcc842bd8a101f6fb8ea27be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41547 } }
I20260929 16:26:58.053836 10649 catalog_manager.cc:5495] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa reported cstate change: term changed from 0 to 1, leader changed from <none> to ec2b1502ebc44bd2ac016cb35d72e6fa (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ec2b1502ebc44bd2ac016cb35d72e6fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3261cc4dc4e4412db5a206497aa6f2e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec2b1502ebc44bd2ac016cb35d72e6fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d725d18bcc842bd8a101f6fb8ea27be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41547 } health_report { overall_health: UNKNOWN } } }
I20260929 16:26:58.066797 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:58.062971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47142 (local address 127.0.98.3:44023)
0929 16:26:58.063109 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 16:26:58.063113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:26:58.063130 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:26:58.063264 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:26:58.063267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:26:58.063321 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:26:58.063408 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:26:58.063689 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:58.064234 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:58.065133 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:26:58.065411 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:26:58.065492 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:26:58.065865 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:26:58.066301 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:26:58.066390 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 16:26:58.066512 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 16:26:58.066567 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:26:58.072801 10795 raft_consensus.cc:1261] T 52f463984566462bb17eb2843ced39b5 P 3261cc4dc4e4412db5a206497aa6f2e3 [term 1 FOLLOWER]: Refusing update from remote peer ec2b1502ebc44bd2ac016cb35d72e6fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:58.072814 10929 raft_consensus.cc:1261] T 52f463984566462bb17eb2843ced39b5 P 8d725d18bcc842bd8a101f6fb8ea27be [term 1 FOLLOWER]: Refusing update from remote peer ec2b1502ebc44bd2ac016cb35d72e6fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:26:58.073638 11132 consensus_queue.cc:1035] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d725d18bcc842bd8a101f6fb8ea27be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:58.074103 11132 consensus_queue.cc:1035] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "3261cc4dc4e4412db5a206497aa6f2e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:26:58.083827 11141 mvcc.cc:205] Tried to move back new op lower bound from 7334703997218471936 to 7334703997137502208. Current Snapshot: MvccSnapshot[applied={T|T < 7334703997218471936}]
I20260929 16:26:58.085520 11139 mvcc.cc:205] Tried to move back new op lower bound from 7334703997218471936 to 7334703997137502208. Current Snapshot: MvccSnapshot[applied={T|T < 7334703997218471936 or (T in {7334703997218471936})}]
I20260929 16:26:58.085747 11140 mvcc.cc:205] Tried to move back new op lower bound from 7334703997218471936 to 7334703997137502208. Current Snapshot: MvccSnapshot[applied={T|T < 7334703997218471936}]
I20260929 16:26:58.092041 10648 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:58.092288 10648 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:26:58.095162 10648 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5aa7c5fe227d406f9f88aa266504a96f: Sending DeleteTablet for 3 replicas of tablet 52f463984566462bb17eb2843ced39b5
I20260929 16:26:58.096015 10775 tablet_service.cc:1514] Processing DeleteTablet for tablet 52f463984566462bb17eb2843ced39b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42406
I20260929 16:26:58.096284 10909 tablet_service.cc:1514] Processing DeleteTablet for tablet 52f463984566462bb17eb2843ced39b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37218
I20260929 16:26:58.096680 11148 tablet_replica.cc:330] T 52f463984566462bb17eb2843ced39b5 P 3261cc4dc4e4412db5a206497aa6f2e3: stopping tablet replica
I20260929 16:26:58.096957 11043 tablet_service.cc:1514] Processing DeleteTablet for tablet 52f463984566462bb17eb2843ced39b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:26:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47134
I20260929 16:26:58.097036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 10712
I20260929 16:26:58.097496 11149 tablet_replica.cc:330] T 52f463984566462bb17eb2843ced39b5 P 8d725d18bcc842bd8a101f6fb8ea27be: stopping tablet replica
I20260929 16:26:58.097896 11149 raft_consensus.cc:2229] T 52f463984566462bb17eb2843ced39b5 P 8d725d18bcc842bd8a101f6fb8ea27be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:26:58.098244 11149 raft_consensus.cc:2258] T 52f463984566462bb17eb2843ced39b5 P 8d725d18bcc842bd8a101f6fb8ea27be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:26:58.099702 11149 ts_tablet_manager.cc:1905] T 52f463984566462bb17eb2843ced39b5 P 8d725d18bcc842bd8a101f6fb8ea27be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:58.102102 11149 ts_tablet_manager.cc:1918] T 52f463984566462bb17eb2843ced39b5 P 8d725d18bcc842bd8a101f6fb8ea27be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:26:58.102222 11149 log.cc:1196] T 52f463984566462bb17eb2843ced39b5 P 8d725d18bcc842bd8a101f6fb8ea27be: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/52f463984566462bb17eb2843ced39b5
I20260929 16:26:58.102530 11149 ts_tablet_manager.cc:1939] T 52f463984566462bb17eb2843ced39b5 P 8d725d18bcc842bd8a101f6fb8ea27be: Deleting consensus metadata
I20260929 16:26:58.103714 10634 catalog_manager.cc:4841] TS 8d725d18bcc842bd8a101f6fb8ea27be (127.0.98.2:41547): tablet 52f463984566462bb17eb2843ced39b5 (table test-table [id=0a36969e91be4ccbb963d0d611a3fcbf]) successfully deleted
I20260929 16:26:58.105433 11150 tablet_replica.cc:330] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa: stopping tablet replica
I20260929 16:26:58.105755 11150 raft_consensus.cc:2229] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:26:58.106166 11150 raft_consensus.cc:2258] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 16:26:58.106784 10636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35531 (error 108)
I20260929 16:26:58.106868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 10846
I20260929 16:26:58.108399 11150 ts_tablet_manager.cc:1905] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:26:58.108919 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:26:58.108532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35531 (local address 127.0.0.1:42414)
0929 16:26:58.108740 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 16:26:58.108815 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260929 16:26:58.109244 10636 catalog_manager.cc:4584] TS 3261cc4dc4e4412db5a206497aa6f2e3 (127.0.98.1:35531): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 52f463984566462bb17eb2843ced39b5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35531: connect: Connection refused (error 111)
I20260929 16:26:58.110921 11150 ts_tablet_manager.cc:1918] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:26:58.111042 11150 log.cc:1196] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/52f463984566462bb17eb2843ced39b5
I20260929 16:26:58.111367 11150 ts_tablet_manager.cc:1939] T 52f463984566462bb17eb2843ced39b5 P ec2b1502ebc44bd2ac016cb35d72e6fa: Deleting consensus metadata
I20260929 16:26:58.112807 10634 catalog_manager.cc:4841] TS ec2b1502ebc44bd2ac016cb35d72e6fa (127.0.98.3:44023): tablet 52f463984566462bb17eb2843ced39b5 (table test-table [id=0a36969e91be4ccbb963d0d611a3fcbf]) successfully deleted
I20260929 16:26:58.116034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 10980
I20260929 16:26:58.121516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 10618
2026-09-29T16:26:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2450 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:26:58 dist-test-slave-vq9v krb5kdc[11154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:26:58 dist-test-slave-vq9v krb5kdc[11154](info): set up 2 sockets
Sep 29 16:26:58 dist-test-slave-vq9v krb5kdc[11154](info): commencing operation
krb5kdc: starting...
W20260929 16:26:59.174237 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:26:59 dist-test-slave-vq9v krb5kdc[11154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:26:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:26:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260929 16:26:59.360970 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task0hb3v3/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/postgres -l logfile start
2026-09-29 16:27:02.199 UTC [11184] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 16:27:02.199 UTC [11184] LOG: listening on IPv4 address "127.0.98.20", port 35511
2026-09-29 16:27:02.203 UTC [11184] LOG: listening on Unix socket "/tmp/.s.PGSQL.35511"
2026-09-29 16:27:02.228 UTC [11187] LOG: database system was shut down at 2026-09-29 16:27:00 UTC
2026-09-29 16:27:02.233 UTC [11184] LOG: database system is ready to accept connections
I20260929 16:27:03.182060 392 mini_postgres.cc:96] Postgres bound to 35511
2026-09-29 16:27:03.188 UTC [11195] FATAL: database "slave" does not exist
127.0.98.20:35511 - accepting connections
I20260929 16:27:03.189070 392 mini_ranger.cc:161] Starting Ranger...
I20260929 16:27:03.204710 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 16:27:03.766999 392 mini_ranger.cc:87] Created ranger Postgres database
I20260929 16:27:03.767102 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ranger-admin
2026-09-29 16:27:04,061 [I] DB FLAVOR :POSTGRES
2026-09-29 16:27:04,062 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:27:04,062 [I] Checking connection..
2026-09-29 16:27:04,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 16:27:04,337 [I] Checking connection passed.
2026-09-29 16:27:04,338 [I] --------- Verifying version history table ---------
2026-09-29 16:27:04,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:27:04,597 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:27:04,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:27:04,843 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:27:04,843 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 16:27:04,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 16:27:05.076 UTC [11259] WARNING: there is no transaction in progress
2026-09-29 16:27:05,083 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 16:27:05,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:27:05,333 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 16:27:05,333 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:27:05,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 16:27:05,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 16:27:05,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vq9v.c.gcp-upstream.internal','N') ;"
2026-09-29 16:27:06,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:27:06,401 [I] Table x_portal_user does not exist in database ranger
2026-09-29 16:27:06,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:27:06,652 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 16:27:06,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:27:06,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:27:07,156 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 16:27:07,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 16:27:08.262 UTC [11367] WARNING: there is no transaction in progress
2026-09-29 16:27:08.282 UTC [11367] WARNING: there is no transaction in progress
2026-09-29 16:27:08.301 UTC [11367] WARNING: there is no transaction in progress
2026-09-29 16:27:08.322 UTC [11367] WARNING: there is no transaction in progress
2026-09-29 16:27:08.342 UTC [11367] WARNING: there is no transaction in progress
2026-09-29 16:27:08.364 UTC [11367] WARNING: there is no transaction in progress
2026-09-29 16:27:08.430 UTC [11367] WARNING: there is no transaction in progress
2026-09-29 16:27:08.440 UTC [11367] WARNING: there is no transaction in progress
2026-09-29 16:27:08.458 UTC [11367] WARNING: there is no transaction in progress
2026-09-29 16:27:08.471 UTC [11367] WARNING: there is no transaction in progress
2026-09-29 16:27:08.483 UTC [11367] WARNING: there is no transaction in progress
2026-09-29 16:27:08.500 UTC [11367] WARNING: there is no transaction in progress
2026-09-29 16:27:08.512 UTC [11367] WARNING: there is no transaction in progress
2026-09-29 16:27:08.519 UTC [11367] WARNING: there is no transaction in progress
2026-09-29 16:27:08.528 UTC [11367] WARNING: there is no transaction in progress
2026-09-29 16:27:08.536 UTC [11367] WARNING: there is no transaction in progress
2026-09-29 16:27:09,350 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 16:27:09,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 16:27:09,591 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 16:27:09,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:27:09,838 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 16:27:09,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:27:10,083 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 16:27:10,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:27:10,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:27:10,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vq9v.c.gcp-upstream.internal';"
2026-09-29 16:27:10,832 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 16:27:10,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 16:27:11,080 [I] DB_PATCHES have already been applied
I20260929 16:27:11.086724 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task0hb3v3/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 16:27:11.086794 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260929 16:27:11.090540 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:60293
Sep 29, 2026 4:27:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:27:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:27:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 4:27:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 4:27:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 4:27:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 4:27:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 16:27:12.219453 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.030s user 0.002s sys 0.009s
Sep 29, 2026 4:27:32 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 4:27:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 4:27:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 4:27:32 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 4:27:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 4:27:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 4:27:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 4:27:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 4:27:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 4:27:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 4:27:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 4:27:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 4:27:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 4:27:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 4:27:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 4:27:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 4:27:32 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 4:27:33 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 16:27:34.044461 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 16:27:34 dist-test-slave-vq9v krb5kdc[11154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699254, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 16:27:34.140240 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 16:27:34.151415 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 16:27:35.176522 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 16:27:35.176642 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ranger-kms
I20260929 16:27:35.176731 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:35511
2026-09-29 16:27:35,368 [I] DB FLAVOR :POSTGRES
2026-09-29 16:27:35,369 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:27:35,369 [I] Checking connection
2026-09-29 16:27:35,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 16:27:35,646 [I] connection success
2026-09-29 16:27:35,646 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 16:27:35,646 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 16:27:35,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 16:27:35,892 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 16:27:35,893 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:27:35,893 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 16:27:35,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35511/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 16:27:36,175 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 16:27:36.663282 392 mini_ranger_kms.cc:310] Created kms service
I20260929 16:27:36.863096 392 mini_ranger_kms.cc:326] Created kudu user
I20260929 16:27:36.910501 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 16:27:36 dist-test-slave-vq9v krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699254, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 16:27:37.339692 392 mini_ranger_kms.cc:384] Added ranger policy
I20260929 16:27:37.340068 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 16:27:37.340121 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260929 16:27:37.343910 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11547
I20260929 16:27:37.344058 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:43533
Sep 29, 2026 4:27:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:27:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:27:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 4:27:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 4:27:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 4:27:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 16:27:39.669857 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.530s user 0.009s sys 0.167s
I20260929 16:27:39.670032 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 16:27:39.670163 392 mini_ranger_kms.cc:401] 127.0.98.20:43533/kms/v1/keys
Sep 29 16:27:39 dist-test-slave-vq9v krb5kdc[11154](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790699259, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 16:27:40 dist-test-slave-vq9v krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699259, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 16:27:40 dist-test-slave-vq9v krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699259, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 16:27:43 dist-test-slave-vq9v krb5kdc[11154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:27:43.205608 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34329
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34329
--ranger_config_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43533/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:27:43.318954 11598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:27:43.319242 11598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:27:43.319281 11598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:27:43.323230 11598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:27:43.323287 11598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:27:43.323307 11598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:27:43.323323 11598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:27:43.323338 11598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:27:43.328823 11598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:43533/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34329
--ranger_config_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:27:43.329844 11598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:27:43.330804 11598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:27:43.339335 11603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:27:43.339349 11604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:27:43.339345 11606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:27:43.339615 11598 server_base.cc:1034] running on GCE node
I20260929 16:27:43.340451 11598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:27:43.341885 11598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:27:43.343091 11598 hybrid_clock.cc:648] HybridClock initialized: now 1790699263343047 us; error 57 us; skew 500 ppm
Sep 29 16:27:43 dist-test-slave-vq9v krb5kdc[11154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:27:43.350991 11598 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:27:43.351999 11598 webserver.cc:460] Webserver started at http://127.0.98.62:38775/ using document root <none> and password file <none>
I20260929 16:27:43.352571 11598 fs_manager.cc:359] Metadata directory not provided
I20260929 16:27:43.352662 11598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:27:43.352910 11598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:27:43 dist-test-slave-vq9v krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 16:27:43.413070 11598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "1870ccd1ecf44112a65763c2051bc076"
format_stamp: "Formatted at 2026-09-29 16:27:43 on dist-test-slave-vq9v"
server_key: "8db7d6b7ec4d7caba1016736aac757f7"
server_key_iv: "4776290f37e8b2d4a206d9b35df76998"
server_key_version: "kuduclusterkey@0"
I20260929 16:27:43.413863 11598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "1870ccd1ecf44112a65763c2051bc076"
format_stamp: "Formatted at 2026-09-29 16:27:43 on dist-test-slave-vq9v"
server_key: "8db7d6b7ec4d7caba1016736aac757f7"
server_key_iv: "4776290f37e8b2d4a206d9b35df76998"
server_key_version: "kuduclusterkey@0"
I20260929 16:27:43.418111 11598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:27:43.433144 11613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:27:43.434718 11598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:27:43.434888 11598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "1870ccd1ecf44112a65763c2051bc076"
format_stamp: "Formatted at 2026-09-29 16:27:43 on dist-test-slave-vq9v"
server_key: "8db7d6b7ec4d7caba1016736aac757f7"
server_key_iv: "4776290f37e8b2d4a206d9b35df76998"
server_key_version: "kuduclusterkey@0"
I20260929 16:27:43.435052 11598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:27:43.455823 11598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:27:43.459261 11598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:27:43.459499 11598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:27:43.468825 11598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34329
I20260929 16:27:43.468868 11665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34329 every 8 connection(s)
I20260929 16:27:43.470009 11598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:27:43.473088 11666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:27:43.473861 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 11598
I20260929 16:27:43.474020 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:27:43.474319 392 external_mini_cluster.cc:1428] Setting key a79dfc9dc66756818b2b4d1c80ed7ddd
I20260929 16:27:43.479971 11666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076: Bootstrap starting.
I20260929 16:27:43.483695 11666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076: Neither blocks nor log segments found. Creating new log.
Sep 29 16:27:43 dist-test-slave-vq9v krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:27:43.484630 11666 log.cc:824] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076: Log is configured to *not* fsync() on all Append() calls
I20260929 16:27:43.487870 11666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076: No bootstrap required, opened a new log
I20260929 16:27:43.492228 11669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:27:43.475830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38086 (local address 127.0.98.62:34329)
0929 16:27:43.476268 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 16:27:43.476284 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:27:43.476319 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:27:43.476944 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:27:43.476961 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:27:43.477380 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:27:43.477669 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:27:43.478736 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:27:43.479949 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:27:43.481288 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:27:43.481833 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:27:43.481974 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:27:43.485279 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:27:43.485294 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:27:43.485310 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:27:43.485344 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:27:43.488354 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:27:43.488946 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:27:43.488953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:27:43.488960 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:27:43.489118 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:27:43.489585 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:27:43.489589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:27:43.489591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:27:43.490221 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:27:43.490426 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 16:27:43.490766 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 16:27:43.491096 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":143,"threads_started":1}
I20260929 16:27:43.492411 11666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1870ccd1ecf44112a65763c2051bc076" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } }
I20260929 16:27:43.492707 11666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:27:43.492786 11666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1870ccd1ecf44112a65763c2051bc076, State: Initialized, Role: FOLLOWER
I20260929 16:27:43.493342 11666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1870ccd1ecf44112a65763c2051bc076" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } }
I20260929 16:27:43.493522 11666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:27:43.493603 11666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:27:43.493702 11666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:27:43.494885 11666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1870ccd1ecf44112a65763c2051bc076" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } }
I20260929 16:27:43.495266 11666 leader_election.cc:302] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1870ccd1ecf44112a65763c2051bc076; no voters:
I20260929 16:27:43.495616 11666 leader_election.cc:288] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:27:43.495764 11671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:27:43.496007 11671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076 [term 1 LEADER]: Becoming Leader. State: Replica: 1870ccd1ecf44112a65763c2051bc076, State: Running, Role: LEADER
I20260929 16:27:43.496371 11671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1870ccd1ecf44112a65763c2051bc076" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } }
I20260929 16:27:43.496937 11666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:27:43.498231 11673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1870ccd1ecf44112a65763c2051bc076. Latest consensus state: current_term: 1 leader_uuid: "1870ccd1ecf44112a65763c2051bc076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1870ccd1ecf44112a65763c2051bc076" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } }
I20260929 16:27:43.498613 11673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076 [sys.catalog]: This master's current role is: LEADER
I20260929 16:27:43.498512 11672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1870ccd1ecf44112a65763c2051bc076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1870ccd1ecf44112a65763c2051bc076" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34329 } } }
I20260929 16:27:43.498955 11672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076 [sys.catalog]: This master's current role is: LEADER
I20260929 16:27:43.499292 11681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:27:43.499558 11666 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 16:27:43.505883 11681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:27:43.513465 11681 catalog_manager.cc:1347] Generated new cluster ID: d5d29c3d40ec4b7d9a8cecd7ad225d78
I20260929 16:27:43.513547 11681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:27:43.550326 11681 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:27:43.551826 11681 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:27:43.557727 11681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076: Generated new TSK 0
I20260929 16:27:43.558666 11681 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 16:27:44.277457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43533/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:27:44.438907 11711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:27:44.439198 11711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:27:44.439280 11711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:27:44.443638 11711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:27:44.443706 11711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:27:44.443809 11711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:27:44.449651 11711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:43533/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:27:44.450909 11711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:27:44.451987 11711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:27:44.463778 11719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:27:44.464385 11716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:27:44.464701 11711 server_base.cc:1034] running on GCE node
W20260929 16:27:44.464987 11717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:27:44.465514 11711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:27:44.466292 11711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:27:44.467495 11711 hybrid_clock.cc:648] HybridClock initialized: now 1790699264467475 us; error 34 us; skew 500 ppm
Sep 29 16:27:44 dist-test-slave-vq9v krb5kdc[11154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:27:44.471263 11711 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:27:44.472383 11711 webserver.cc:460] Webserver started at http://127.0.98.1:45345/ using document root <none> and password file <none>
I20260929 16:27:44.473135 11711 fs_manager.cc:359] Metadata directory not provided
I20260929 16:27:44.473331 11711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:27:44.473618 11711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:27:44 dist-test-slave-vq9v krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 16:27:44.495098 11711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "d5ebccc255f349f5be67731ae451bf8b"
format_stamp: "Formatted at 2026-09-29 16:27:44 on dist-test-slave-vq9v"
server_key: "dab6bf32d797acbf494c5f18f1f3bb98"
server_key_iv: "8a436efe4772a4b571e39ab11288dd58"
server_key_version: "kuduclusterkey@0"
I20260929 16:27:44.495982 11711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d5ebccc255f349f5be67731ae451bf8b"
format_stamp: "Formatted at 2026-09-29 16:27:44 on dist-test-slave-vq9v"
server_key: "dab6bf32d797acbf494c5f18f1f3bb98"
server_key_iv: "8a436efe4772a4b571e39ab11288dd58"
server_key_version: "kuduclusterkey@0"
I20260929 16:27:44.501633 11711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260929 16:27:44.524923 11728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:27:44.527038 11711 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.000s
I20260929 16:27:44.527225 11711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "d5ebccc255f349f5be67731ae451bf8b"
format_stamp: "Formatted at 2026-09-29 16:27:44 on dist-test-slave-vq9v"
server_key: "dab6bf32d797acbf494c5f18f1f3bb98"
server_key_iv: "8a436efe4772a4b571e39ab11288dd58"
server_key_version: "kuduclusterkey@0"
I20260929 16:27:44.527441 11711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:27:44.568604 11711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:27:44.572526 11711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:27:44.572757 11711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:27:44.573581 11711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:27:44.574707 11711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:27:44.574779 11711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:27:44.574846 11711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:27:44.574873 11711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:27:44.587437 11711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34865
I20260929 16:27:44.587618 11842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34865 every 8 connection(s)
I20260929 16:27:44.588729 11711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:27:44.589950 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 11711
I20260929 16:27:44.590063 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:27:44.590454 392 external_mini_cluster.cc:1428] Setting key f09c9518fdbd869563667532dbd991b2
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 29 16:27:44 dist-test-slave-vq9v krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:27:44.631649 11846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:27:44.593402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43333 (local address 127.0.98.62:34329)
0929 16:27:44.597461 (+ 4059us) server_negotiation.cc:206] Beginning negotiation
0929 16:27:44.597466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:27:44.597732 (+ 266us) server_negotiation.cc:407] Connection header received
0929 16:27:44.599856 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:27:44.599860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:27:44.599923 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:27:44.600005 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:27:44.617393 (+ 17388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:27:44.618289 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:27:44.620304 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:27:44.620683 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:27:44.620775 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:27:44.626643 (+ 5868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:27:44.626648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:27:44.626651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:27:44.626686 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:27:44.628762 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:27:44.629586 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:27:44.629590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:27:44.629592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:27:44.629692 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:27:44.630136 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:27:44.630139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:27:44.630141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:27:44.630411 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:27:44.630505 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 16:27:44.631302 (+ 797us) server_negotiation.cc:299] Negotiation successful
0929 16:27:44.631448 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3047,"thread_start_us":83,"threads_started":1}
I20260929 16:27:44.634140 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:27:44.593087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34329 (local address 127.0.98.1:43333)
0929 16:27:44.597509 (+ 4422us) negotiation.cc:107] Waiting for socket to connect
0929 16:27:44.597605 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0929 16:27:44.599479 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:27:44.611717 (+ 12238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:27:44.611731 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:27:44.612185 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:27:44.613038 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:27:44.613063 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:27:44.618514 (+ 5451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:27:44.618518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:27:44.620112 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:27:44.620129 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:27:44.621024 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:27:44.621030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:27:44.621452 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:27:44.622212 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:27:44.622364 (+ 152us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:27:44.626369 (+ 4005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:27:44.628947 (+ 2578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:27:44.628954 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:27:44.628976 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:27:44.629442 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:27:44.629816 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:27:44.629819 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:27:44.629821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:27:44.630021 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:27:44.630575 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:27:44.630583 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:27:44.630972 (+ 389us) client_negotiation.cc:769] Sending connection context
0929 16:27:44.632173 (+ 1201us) client_negotiation.cc:240] Negotiation successful
0929 16:27:44.632706 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4011,"thread_start_us":156,"threads_started":1}
I20260929 16:27:44.636042 11843 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34329
I20260929 16:27:44.636322 11843 heartbeater.cc:464] Registering TS with master...
I20260929 16:27:44.636904 11843 heartbeater.cc:511] Master 127.0.98.62:34329 requested a full tablet report, sending...
I20260929 16:27:44.638813 11630 ts_manager.cc:194] Registered new tserver with Master: d5ebccc255f349f5be67731ae451bf8b (127.0.98.1:34865)
I20260929 16:27:44.640453 11630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 16:27:44.662283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43533/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:27:44 dist-test-slave-vq9v krb5kdc[11154](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790699264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 16:27:44.827940 11850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:27:44.828312 11850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:27:44.828375 11850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:27:44.834403 11850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:27:44.834489 11850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:27:44.834575 11850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:27:44.842525 11850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:43533/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:27:44.844010 11850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:27:44.845301 11850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:27:44.858131 11857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:27:44.858532 11859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:27:44.859632 11850 server_base.cc:1034] running on GCE node
W20260929 16:27:44.859944 11856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:27:44.860437 11850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:27:44.861248 11850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:27:44.862432 11850 hybrid_clock.cc:648] HybridClock initialized: now 1790699264862400 us; error 43 us; skew 500 ppm
Sep 29 16:27:44 dist-test-slave-vq9v krb5kdc[11154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:27:44.867305 11850 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:27:44.868573 11850 webserver.cc:460] Webserver started at http://127.0.98.2:42607/ using document root <none> and password file <none>
I20260929 16:27:44.870018 11850 fs_manager.cc:359] Metadata directory not provided
I20260929 16:27:44.870186 11850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:27:44.870592 11850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:27:44 dist-test-slave-vq9v krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 16:27:44.889909 11850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "9b64044f9b894140879afaec866e1092"
format_stamp: "Formatted at 2026-09-29 16:27:44 on dist-test-slave-vq9v"
server_key: "0953a802a0f3908fc07fc0ac4ea78ce3"
server_key_iv: "40266a3df07b89658cb39a949c96f43f"
server_key_version: "kuduclusterkey@0"
I20260929 16:27:44.892268 11850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9b64044f9b894140879afaec866e1092"
format_stamp: "Formatted at 2026-09-29 16:27:44 on dist-test-slave-vq9v"
server_key: "0953a802a0f3908fc07fc0ac4ea78ce3"
server_key_iv: "40266a3df07b89658cb39a949c96f43f"
server_key_version: "kuduclusterkey@0"
I20260929 16:27:44.898286 11850 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:27:44.937016 11866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:27:44.938942 11850 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260929 16:27:44.939121 11850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "9b64044f9b894140879afaec866e1092"
format_stamp: "Formatted at 2026-09-29 16:27:44 on dist-test-slave-vq9v"
server_key: "0953a802a0f3908fc07fc0ac4ea78ce3"
server_key_iv: "40266a3df07b89658cb39a949c96f43f"
server_key_version: "kuduclusterkey@0"
I20260929 16:27:44.939298 11850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:27:44.976621 11850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:27:44.980806 11850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:27:44.981033 11850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:27:44.981812 11850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:27:44.983011 11850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:27:44.983089 11850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:27:44.983157 11850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:27:44.983189 11850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:27:44.999179 11850 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33167
I20260929 16:27:45.000576 11850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:27:45.002497 11979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33167 every 8 connection(s)
I20260929 16:27:45.002545 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 11850
I20260929 16:27:45.002669 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:27:45.002985 392 external_mini_cluster.cc:1428] Setting key 237982288ad9baa5ea55ea86648da6c9
Sep 29 16:27:45 dist-test-slave-vq9v krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 16:27:45.022637 11846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:27:45.006156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36409 (local address 127.0.98.62:34329)
0929 16:27:45.006284 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 16:27:45.006289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:27:45.007154 (+ 865us) server_negotiation.cc:407] Connection header received
0929 16:27:45.008547 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:27:45.008550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:27:45.008609 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:27:45.008667 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:27:45.010453 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.011334 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.013822 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.014198 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.014331 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:27:45.017478 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:27:45.017482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:27:45.017485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:27:45.017521 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:27:45.019452 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:27:45.020135 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:27:45.020139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:27:45.020141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:27:45.020249 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:27:45.020683 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:27:45.020689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:27:45.020691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:27:45.020933 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:27:45.021020 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 16:27:45.021852 (+ 832us) server_negotiation.cc:299] Negotiation successful
0929 16:27:45.022420 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 16:27:45.024348 11982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:27:45.004949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34329 (local address 127.0.98.2:36409)
0929 16:27:45.006969 (+ 2020us) negotiation.cc:107] Waiting for socket to connect
0929 16:27:45.007040 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 16:27:45.008279 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:27:45.009130 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:27:45.009139 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:27:45.009551 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:27:45.010264 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:27:45.010285 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.012161 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.012165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:27:45.013632 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:27:45.013644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.014402 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.014406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:27:45.014643 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:27:45.015406 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:27:45.015431 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:27:45.017244 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:27:45.019609 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:27:45.019617 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:27:45.019638 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:27:45.020006 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:27:45.020364 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:27:45.020369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:27:45.020371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:27:45.020570 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:27:45.021092 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:27:45.021099 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:27:45.021536 (+ 437us) client_negotiation.cc:769] Sending connection context
0929 16:27:45.022574 (+ 1038us) client_negotiation.cc:240] Negotiation successful
0929 16:27:45.023216 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1777,"thread_start_us":193,"threads_started":1}
I20260929 16:27:45.025866 11980 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34329
I20260929 16:27:45.026151 11980 heartbeater.cc:464] Registering TS with master...
I20260929 16:27:45.026770 11980 heartbeater.cc:511] Master 127.0.98.62:34329 requested a full tablet report, sending...
I20260929 16:27:45.027923 11630 ts_manager.cc:194] Registered new tserver with Master: 9b64044f9b894140879afaec866e1092 (127.0.98.2:33167)
I20260929 16:27:45.028640 11630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 16:27:45.045926 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43533/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:27:45.225407 11985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:27:45.225809 11985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:27:45.225872 11985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Sep 29 16:27:45 dist-test-slave-vq9v krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260929 16:27:45.232043 11985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:27:45.232138 11985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:27:45.232226 11985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:27:45.240666 11985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:43533/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:27:45.242113 11985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:27:45.243409 11985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:27:45.266163 11993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:27:45.266158 11991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:27:45.267300 11990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:27:45.270012 11985 server_base.cc:1034] running on GCE node
I20260929 16:27:45.270699 11985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:27:45.271549 11985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:27:45.279223 11985 hybrid_clock.cc:648] HybridClock initialized: now 1790699265279134 us; error 114 us; skew 500 ppm
Sep 29 16:27:45 dist-test-slave-vq9v krb5kdc[11154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:27:45.283143 11985 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:27:45.284380 11985 webserver.cc:460] Webserver started at http://127.0.98.3:35129/ using document root <none> and password file <none>
I20260929 16:27:45.285193 11985 fs_manager.cc:359] Metadata directory not provided
I20260929 16:27:45.285368 11985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:27:45.285640 11985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:27:45 dist-test-slave-vq9v krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 16:27:45.302474 11985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "a324b52a8cac494380b9fb79fe2a8e01"
format_stamp: "Formatted at 2026-09-29 16:27:45 on dist-test-slave-vq9v"
server_key: "3ff23d4bdd2f89fd774ae62bef79b1a1"
server_key_iv: "8e970547c092129c7b18e05fa0d3afe2"
server_key_version: "kuduclusterkey@0"
I20260929 16:27:45.303539 11985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a324b52a8cac494380b9fb79fe2a8e01"
format_stamp: "Formatted at 2026-09-29 16:27:45 on dist-test-slave-vq9v"
server_key: "3ff23d4bdd2f89fd774ae62bef79b1a1"
server_key_iv: "8e970547c092129c7b18e05fa0d3afe2"
server_key_version: "kuduclusterkey@0"
I20260929 16:27:45.310811 11985 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260929 16:27:45.332346 12000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:27:45.334323 11985 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.001s
I20260929 16:27:45.334478 11985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "a324b52a8cac494380b9fb79fe2a8e01"
format_stamp: "Formatted at 2026-09-29 16:27:45 on dist-test-slave-vq9v"
server_key: "3ff23d4bdd2f89fd774ae62bef79b1a1"
server_key_iv: "8e970547c092129c7b18e05fa0d3afe2"
server_key_version: "kuduclusterkey@0"
I20260929 16:27:45.334640 11985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:27:45.380004 11985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:27:45.383733 11985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:27:45.383955 11985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:27:45.384711 11985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:27:45.385831 11985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:27:45.385907 11985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:27:45.386009 11985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:27:45.386055 11985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:27:45.398281 11985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44447
I20260929 16:27:45.398252 12113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44447 every 8 connection(s)
I20260929 16:27:45.399822 11985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:27:45.404932 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 11985
I20260929 16:27:45.405061 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:27:45.405454 392 external_mini_cluster.cc:1428] Setting key 15d81761f705a3d75d60cc01c5539b8b
Sep 29 16:27:45 dist-test-slave-vq9v krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:27:45.418064 11846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:27:45.403871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39069 (local address 127.0.98.62:34329)
0929 16:27:45.404007 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 16:27:45.404012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:27:45.404033 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:27:45.404738 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:27:45.404742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:27:45.404801 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:27:45.404854 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:27:45.406657 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.407498 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.409390 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.409771 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.409928 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:27:45.413054 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:27:45.413058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:27:45.413061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:27:45.413095 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:27:45.415142 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:27:45.415895 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:27:45.415900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:27:45.415902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:27:45.416018 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:27:45.416464 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:27:45.416468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:27:45.416471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:27:45.416789 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:27:45.416899 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:27:45.417738 (+ 839us) server_negotiation.cc:299] Negotiation successful
0929 16:27:45.417884 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:27:45.419093 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:27:45.402658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34329 (local address 127.0.98.3:39069)
0929 16:27:45.403272 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0929 16:27:45.403335 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 16:27:45.404418 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:27:45.405348 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:27:45.405357 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:27:45.405736 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:27:45.406479 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:27:45.406505 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.407675 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.407678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:27:45.409169 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:27:45.409183 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.409990 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.409994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:27:45.410218 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:27:45.411035 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:27:45.411057 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:27:45.412774 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:27:45.415310 (+ 2536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:27:45.415319 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:27:45.415340 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:27:45.415765 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:27:45.416127 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:27:45.416130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:27:45.416132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:27:45.416349 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:27:45.416915 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:27:45.416923 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:27:45.417354 (+ 431us) client_negotiation.cc:769] Sending connection context
0929 16:27:45.417610 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 16:27:45.417991 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":160,"threads_started":1}
I20260929 16:27:45.421010 12114 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34329
I20260929 16:27:45.421749 12114 heartbeater.cc:464] Registering TS with master...
I20260929 16:27:45.422464 12114 heartbeater.cc:511] Master 127.0.98.62:34329 requested a full tablet report, sending...
I20260929 16:27:45.423712 11630 ts_manager.cc:194] Registered new tserver with Master: a324b52a8cac494380b9fb79fe2a8e01 (127.0.98.3:44447)
I20260929 16:27:45.424527 11630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39069
I20260929 16:27:45.432746 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:27:45.445195 11846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:27:45.435140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38098 (local address 127.0.98.62:34329)
0929 16:27:45.435306 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 16:27:45.435311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:27:45.435734 (+ 423us) server_negotiation.cc:407] Connection header received
0929 16:27:45.435864 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:27:45.435867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:27:45.435921 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:27:45.436007 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:27:45.436935 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.437786 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.438984 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.439371 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.439490 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:27:45.441358 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:27:45.441362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:27:45.441366 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:27:45.441397 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:27:45.443287 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:27:45.443781 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:27:45.443784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:27:45.443786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:27:45.443886 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:27:45.444209 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:27:45.444212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:27:45.444215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:27:45.444472 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:27:45.444643 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 16:27:45.444890 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 16:27:45.445022 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:27:45.449100 11630 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:27:45.453392 11630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:27:45.475184 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:27:45.464711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34865 (local address 127.0.0.1:40566)
0929 16:27:45.467196 (+ 2485us) negotiation.cc:107] Waiting for socket to connect
0929 16:27:45.467263 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 16:27:45.467388 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:27:45.468432 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:27:45.468435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:27:45.468454 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:27:45.468595 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:27:45.468603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.470904 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.470910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:27:45.472356 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:27:45.472367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.474613 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.474617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:27:45.474740 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:27:45.474782 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 16:27:45.474891 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 16:27:45.475020 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2298,"mutex_wait_us":15,"thread_start_us":116,"threads_started":1}
I20260929 16:27:45.475878 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:27:45.464991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.0.98.1:34865)
0929 16:27:45.467795 (+ 2804us) server_negotiation.cc:206] Beginning negotiation
0929 16:27:45.467802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:27:45.467826 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:27:45.467900 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:27:45.467905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:27:45.468119 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:27:45.468276 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:27:45.468743 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.470767 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.472509 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.473475 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.473594 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:27:45.475547 (+ 1953us) server_negotiation.cc:1035] Waiting for connection context
0929 16:27:45.475635 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 16:27:45.475740 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2666,"thread_start_us":74,"threads_started":1}
I20260929 16:27:45.479617 11759 tablet_service.cc:1467] Processing CreateTablet for tablet af1b6ce82a1446da882212a070c823b4 (DEFAULT_TABLE table=test-table [id=7a1578ce84464faa9b8610b36a6ff776]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:27:45.480822 11759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af1b6ce82a1446da882212a070c823b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:27:45.484508 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:27:45.465895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33167 (local address 127.0.0.1:59542)
0929 16:27:45.467394 (+ 1499us) negotiation.cc:107] Waiting for socket to connect
0929 16:27:45.467408 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:27:45.467503 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:27:45.468367 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:27:45.468373 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:27:45.468398 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:27:45.468587 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:27:45.468594 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.481707 (+ 13113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.481711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:27:45.482874 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:27:45.482884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.484035 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.484039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:27:45.484152 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:27:45.484182 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:27:45.484269 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 16:27:45.484351 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1429,"mutex_wait_us":27,"thread_start_us":76,"threads_started":1}
I20260929 16:27:45.485375 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:27:45.466402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59542 (local address 127.0.98.2:33167)
0929 16:27:45.467652 (+ 1250us) server_negotiation.cc:206] Beginning negotiation
0929 16:27:45.467666 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:27:45.467691 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:27:45.467767 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:27:45.467772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:27:45.467989 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:27:45.468715 (+ 726us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:27:45.468837 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.481592 (+ 12755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.483050 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.483887 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.484853 (+ 966us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:27:45.485018 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 16:27:45.485075 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 16:27:45.485175 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1147,"thread_start_us":88,"threads_started":1}
I20260929 16:27:45.486492 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:27:45.466182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44447 (local address 127.0.0.1:45640)
0929 16:27:45.476210 (+ 10028us) negotiation.cc:107] Waiting for socket to connect
0929 16:27:45.476225 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:27:45.476328 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:27:45.479715 (+ 3387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:27:45.479720 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:27:45.479739 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:27:45.479831 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:27:45.479838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.482233 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.482237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:27:45.483393 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:27:45.483404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.486094 (+ 2690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.486098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:27:45.486192 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:27:45.486217 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:27:45.486283 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 16:27:45.486356 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9967,"thread_start_us":95,"threads_started":1}
I20260929 16:27:45.487362 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:27:45.466720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45640 (local address 127.0.98.3:44447)
0929 16:27:45.468263 (+ 1543us) server_negotiation.cc:206] Beginning negotiation
0929 16:27:45.468274 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:27:45.477385 (+ 9111us) server_negotiation.cc:407] Connection header received
0929 16:27:45.477484 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:27:45.477501 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:27:45.477742 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:27:45.477904 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:27:45.479973 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.482097 (+ 2124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.483570 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.484372 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.484481 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:27:45.487056 (+ 2575us) server_negotiation.cc:1035] Waiting for connection context
0929 16:27:45.487113 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 16:27:45.487216 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1439,"thread_start_us":100,"threads_started":1}
I20260929 16:27:45.488571 11914 tablet_service.cc:1467] Processing CreateTablet for tablet af1b6ce82a1446da882212a070c823b4 (DEFAULT_TABLE table=test-table [id=7a1578ce84464faa9b8610b36a6ff776]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:27:45.489814 11914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af1b6ce82a1446da882212a070c823b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:27:45.490259 12132 tablet_bootstrap.cc:492] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b: Bootstrap starting.
I20260929 16:27:45.492718 12132 tablet_bootstrap.cc:654] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b: Neither blocks nor log segments found. Creating new log.
I20260929 16:27:45.493741 12132 log.cc:824] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:27:45.496026 12132 tablet_bootstrap.cc:492] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b: No bootstrap required, opened a new log
I20260929 16:27:45.496245 12132 ts_tablet_manager.cc:1397] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:27:45.496419 12048 tablet_service.cc:1467] Processing CreateTablet for tablet af1b6ce82a1446da882212a070c823b4 (DEFAULT_TABLE table=test-table [id=7a1578ce84464faa9b8610b36a6ff776]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:27:45.497758 12048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af1b6ce82a1446da882212a070c823b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:27:45.500676 12132 raft_consensus.cc:348] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ebccc255f349f5be67731ae451bf8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34865 } } peers { permanent_uuid: "9b64044f9b894140879afaec866e1092" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33167 } } peers { permanent_uuid: "a324b52a8cac494380b9fb79fe2a8e01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44447 } }
I20260929 16:27:45.500972 12132 raft_consensus.cc:374] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:27:45.501025 12132 raft_consensus.cc:729] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ebccc255f349f5be67731ae451bf8b, State: Initialized, Role: FOLLOWER
I20260929 16:27:45.501576 12132 consensus_queue.cc:260] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ebccc255f349f5be67731ae451bf8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34865 } } peers { permanent_uuid: "9b64044f9b894140879afaec866e1092" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33167 } } peers { permanent_uuid: "a324b52a8cac494380b9fb79fe2a8e01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44447 } }
I20260929 16:27:45.502673 12132 ts_tablet_manager.cc:1428] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:27:45.503082 11843 heartbeater.cc:503] Master 127.0.98.62:34329 was elected leader, sending a full tablet report...
I20260929 16:27:45.509327 12135 tablet_bootstrap.cc:492] T af1b6ce82a1446da882212a070c823b4 P a324b52a8cac494380b9fb79fe2a8e01: Bootstrap starting.
I20260929 16:27:45.511333 12136 tablet_bootstrap.cc:492] T af1b6ce82a1446da882212a070c823b4 P 9b64044f9b894140879afaec866e1092: Bootstrap starting.
I20260929 16:27:45.512209 12135 tablet_bootstrap.cc:654] T af1b6ce82a1446da882212a070c823b4 P a324b52a8cac494380b9fb79fe2a8e01: Neither blocks nor log segments found. Creating new log.
I20260929 16:27:45.513640 12135 log.cc:824] T af1b6ce82a1446da882212a070c823b4 P a324b52a8cac494380b9fb79fe2a8e01: Log is configured to *not* fsync() on all Append() calls
I20260929 16:27:45.514557 12136 tablet_bootstrap.cc:654] T af1b6ce82a1446da882212a070c823b4 P 9b64044f9b894140879afaec866e1092: Neither blocks nor log segments found. Creating new log.
I20260929 16:27:45.515904 12136 log.cc:824] T af1b6ce82a1446da882212a070c823b4 P 9b64044f9b894140879afaec866e1092: Log is configured to *not* fsync() on all Append() calls
I20260929 16:27:45.515970 12135 tablet_bootstrap.cc:492] T af1b6ce82a1446da882212a070c823b4 P a324b52a8cac494380b9fb79fe2a8e01: No bootstrap required, opened a new log
I20260929 16:27:45.516153 12135 ts_tablet_manager.cc:1397] T af1b6ce82a1446da882212a070c823b4 P a324b52a8cac494380b9fb79fe2a8e01: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260929 16:27:45.518014 12136 tablet_bootstrap.cc:492] T af1b6ce82a1446da882212a070c823b4 P 9b64044f9b894140879afaec866e1092: No bootstrap required, opened a new log
I20260929 16:27:45.518234 12136 ts_tablet_manager.cc:1397] T af1b6ce82a1446da882212a070c823b4 P 9b64044f9b894140879afaec866e1092: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 16:27:45.520321 12135 raft_consensus.cc:348] T af1b6ce82a1446da882212a070c823b4 P a324b52a8cac494380b9fb79fe2a8e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ebccc255f349f5be67731ae451bf8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34865 } } peers { permanent_uuid: "9b64044f9b894140879afaec866e1092" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33167 } } peers { permanent_uuid: "a324b52a8cac494380b9fb79fe2a8e01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44447 } }
I20260929 16:27:45.520622 12135 raft_consensus.cc:374] T af1b6ce82a1446da882212a070c823b4 P a324b52a8cac494380b9fb79fe2a8e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:27:45.520684 12135 raft_consensus.cc:729] T af1b6ce82a1446da882212a070c823b4 P a324b52a8cac494380b9fb79fe2a8e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a324b52a8cac494380b9fb79fe2a8e01, State: Initialized, Role: FOLLOWER
I20260929 16:27:45.521255 12135 consensus_queue.cc:260] T af1b6ce82a1446da882212a070c823b4 P a324b52a8cac494380b9fb79fe2a8e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ebccc255f349f5be67731ae451bf8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34865 } } peers { permanent_uuid: "9b64044f9b894140879afaec866e1092" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33167 } } peers { permanent_uuid: "a324b52a8cac494380b9fb79fe2a8e01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44447 } }
I20260929 16:27:45.522339 12135 ts_tablet_manager.cc:1428] T af1b6ce82a1446da882212a070c823b4 P a324b52a8cac494380b9fb79fe2a8e01: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:27:45.522250 12136 raft_consensus.cc:348] T af1b6ce82a1446da882212a070c823b4 P 9b64044f9b894140879afaec866e1092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ebccc255f349f5be67731ae451bf8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34865 } } peers { permanent_uuid: "9b64044f9b894140879afaec866e1092" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33167 } } peers { permanent_uuid: "a324b52a8cac494380b9fb79fe2a8e01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44447 } }
I20260929 16:27:45.522509 12136 raft_consensus.cc:374] T af1b6ce82a1446da882212a070c823b4 P 9b64044f9b894140879afaec866e1092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:27:45.522574 12136 raft_consensus.cc:729] T af1b6ce82a1446da882212a070c823b4 P 9b64044f9b894140879afaec866e1092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b64044f9b894140879afaec866e1092, State: Initialized, Role: FOLLOWER
I20260929 16:27:45.522733 12114 heartbeater.cc:503] Master 127.0.98.62:34329 was elected leader, sending a full tablet report...
I20260929 16:27:45.523047 12136 consensus_queue.cc:260] T af1b6ce82a1446da882212a070c823b4 P 9b64044f9b894140879afaec866e1092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ebccc255f349f5be67731ae451bf8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34865 } } peers { permanent_uuid: "9b64044f9b894140879afaec866e1092" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33167 } } peers { permanent_uuid: "a324b52a8cac494380b9fb79fe2a8e01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44447 } }
I20260929 16:27:45.524017 12136 ts_tablet_manager.cc:1428] T af1b6ce82a1446da882212a070c823b4 P 9b64044f9b894140879afaec866e1092: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:27:45.524369 11980 heartbeater.cc:503] Master 127.0.98.62:34329 was elected leader, sending a full tablet report...
Sep 29 16:27:45 dist-test-slave-vq9v krb5kdc[11154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260929 16:27:45.590781 11844 tablet.cc:2369] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:27:45.639367 12134 raft_consensus.cc:482] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:27:45.639604 12134 raft_consensus.cc:504] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ebccc255f349f5be67731ae451bf8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34865 } } peers { permanent_uuid: "9b64044f9b894140879afaec866e1092" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33167 } } peers { permanent_uuid: "a324b52a8cac494380b9fb79fe2a8e01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44447 } }
I20260929 16:27:45.641081 12134 leader_election.cc:288] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b64044f9b894140879afaec866e1092 (127.0.98.2:33167), a324b52a8cac494380b9fb79fe2a8e01 (127.0.98.3:44447)
I20260929 16:27:45.647294 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:27:45.642218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55213 (local address 127.0.98.2:33167)
0929 16:27:45.642491 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 16:27:45.642495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:27:45.642602 (+ 107us) server_negotiation.cc:407] Connection header received
0929 16:27:45.642847 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:27:45.642851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:27:45.642903 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:27:45.642953 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:27:45.643379 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.644155 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.645886 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.646516 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.646605 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:27:45.646698 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 16:27:45.647063 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 16:27:45.647152 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260929 16:27:45.647778 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:27:45.641495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33167 (local address 127.0.98.1:55213)
0929 16:27:45.642507 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0929 16:27:45.642525 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:27:45.642710 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:27:45.643100 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:27:45.643103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:27:45.643121 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:27:45.643225 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:27:45.643233 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.644309 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.644312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:27:45.645719 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:27:45.645731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.646803 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.646807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:27:45.646906 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:27:45.646931 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:27:45.647529 (+ 598us) client_negotiation.cc:240] Negotiation successful
0929 16:27:45.647619 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"thread_start_us":127,"threads_started":1}
I20260929 16:27:45.648525 11930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1b6ce82a1446da882212a070c823b4" candidate_uuid: "d5ebccc255f349f5be67731ae451bf8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b64044f9b894140879afaec866e1092" is_pre_election: true
I20260929 16:27:45.648926 11930 raft_consensus.cc:2454] T af1b6ce82a1446da882212a070c823b4 P 9b64044f9b894140879afaec866e1092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5ebccc255f349f5be67731ae451bf8b in term 0.
I20260929 16:27:45.649499 11731 leader_election.cc:302] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b64044f9b894140879afaec866e1092, d5ebccc255f349f5be67731ae451bf8b; no voters:
I20260929 16:27:45.649855 12134 raft_consensus.cc:2781] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:27:45.649935 12134 raft_consensus.cc:482] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:27:45.649978 12134 raft_consensus.cc:3037] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:27:45.651337 12134 raft_consensus.cc:504] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ebccc255f349f5be67731ae451bf8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34865 } } peers { permanent_uuid: "9b64044f9b894140879afaec866e1092" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33167 } } peers { permanent_uuid: "a324b52a8cac494380b9fb79fe2a8e01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44447 } }
I20260929 16:27:45.651741 12134 leader_election.cc:288] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b [CANDIDATE]: Term 1 election: Requested vote from peers 9b64044f9b894140879afaec866e1092 (127.0.98.2:33167), a324b52a8cac494380b9fb79fe2a8e01 (127.0.98.3:44447)
W20260929 16:27:45.652064 12115 tablet.cc:2369] T af1b6ce82a1446da882212a070c823b4 P a324b52a8cac494380b9fb79fe2a8e01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:27:45.652354 11930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1b6ce82a1446da882212a070c823b4" candidate_uuid: "d5ebccc255f349f5be67731ae451bf8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b64044f9b894140879afaec866e1092"
I20260929 16:27:45.652503 11930 raft_consensus.cc:3037] T af1b6ce82a1446da882212a070c823b4 P 9b64044f9b894140879afaec866e1092 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:27:45.652979 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:27:45.642583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40783 (local address 127.0.98.3:44447)
0929 16:27:45.642710 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 16:27:45.642715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:27:45.648431 (+ 5716us) server_negotiation.cc:407] Connection header received
0929 16:27:45.648650 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:27:45.648654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:27:45.648709 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:27:45.648760 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:27:45.649265 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.649957 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.651423 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.652015 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.652096 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:27:45.652160 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 16:27:45.652713 (+ 553us) server_negotiation.cc:299] Negotiation successful
0929 16:27:45.652846 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 16:27:45.653970 11930 raft_consensus.cc:2454] T af1b6ce82a1446da882212a070c823b4 P 9b64044f9b894140879afaec866e1092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ebccc255f349f5be67731ae451bf8b in term 1.
I20260929 16:27:45.654259 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:27:45.642136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44447 (local address 127.0.98.1:40783)
0929 16:27:45.648348 (+ 6212us) negotiation.cc:107] Waiting for socket to connect
0929 16:27:45.648364 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:27:45.648510 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:27:45.648934 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:27:45.648951 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:27:45.648972 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:27:45.649093 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:27:45.649100 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.650127 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.650131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:27:45.651246 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:27:45.651260 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.652447 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.652453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:27:45.652544 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:27:45.652568 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:27:45.653994 (+ 1426us) client_negotiation.cc:240] Negotiation successful
0929 16:27:45.654084 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6149,"thread_start_us":93,"threads_started":1}
I20260929 16:27:45.654405 11731 leader_election.cc:302] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b64044f9b894140879afaec866e1092, d5ebccc255f349f5be67731ae451bf8b; no voters:
I20260929 16:27:45.654635 12134 raft_consensus.cc:2781] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:27:45.654863 12134 raft_consensus.cc:686] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b [term 1 LEADER]: Becoming Leader. State: Replica: d5ebccc255f349f5be67731ae451bf8b, State: Running, Role: LEADER
I20260929 16:27:45.655243 12134 consensus_queue.cc:237] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ebccc255f349f5be67731ae451bf8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34865 } } peers { permanent_uuid: "9b64044f9b894140879afaec866e1092" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33167 } } peers { permanent_uuid: "a324b52a8cac494380b9fb79fe2a8e01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44447 } }
I20260929 16:27:45.655369 12068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1b6ce82a1446da882212a070c823b4" candidate_uuid: "d5ebccc255f349f5be67731ae451bf8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a324b52a8cac494380b9fb79fe2a8e01" is_pre_election: true
I20260929 16:27:45.655731 12068 raft_consensus.cc:2454] T af1b6ce82a1446da882212a070c823b4 P a324b52a8cac494380b9fb79fe2a8e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5ebccc255f349f5be67731ae451bf8b in term 0.
I20260929 16:27:45.656096 12067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1b6ce82a1446da882212a070c823b4" candidate_uuid: "d5ebccc255f349f5be67731ae451bf8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a324b52a8cac494380b9fb79fe2a8e01"
I20260929 16:27:45.656234 12067 raft_consensus.cc:3037] T af1b6ce82a1446da882212a070c823b4 P a324b52a8cac494380b9fb79fe2a8e01 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:27:45.657735 12067 raft_consensus.cc:2454] T af1b6ce82a1446da882212a070c823b4 P a324b52a8cac494380b9fb79fe2a8e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ebccc255f349f5be67731ae451bf8b in term 1.
I20260929 16:27:45.659044 11630 catalog_manager.cc:5495] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b reported cstate change: term changed from 0 to 1, leader changed from <none> to d5ebccc255f349f5be67731ae451bf8b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d5ebccc255f349f5be67731ae451bf8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ebccc255f349f5be67731ae451bf8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b64044f9b894140879afaec866e1092" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a324b52a8cac494380b9fb79fe2a8e01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44447 } health_report { overall_health: UNKNOWN } } }
I20260929 16:27:45.710117 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:27:45.704749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40580 (local address 127.0.98.1:34865)
0929 16:27:45.704877 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 16:27:45.704880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:27:45.704895 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:27:45.705109 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:27:45.705113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:27:45.705185 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:27:45.705321 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:27:45.705790 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.706587 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.707832 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:27:45.708166 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:27:45.708268 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:27:45.708722 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:27:45.709319 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:27:45.709434 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 16:27:45.709911 (+ 477us) server_negotiation.cc:299] Negotiation successful
0929 16:27:45.709975 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 16:27:45.717449 12067 raft_consensus.cc:1261] T af1b6ce82a1446da882212a070c823b4 P a324b52a8cac494380b9fb79fe2a8e01 [term 1 FOLLOWER]: Refusing update from remote peer d5ebccc255f349f5be67731ae451bf8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:27:45.718827 12145 consensus_queue.cc:1035] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a324b52a8cac494380b9fb79fe2a8e01" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 16:27:45.724855 11930 raft_consensus.cc:1261] T af1b6ce82a1446da882212a070c823b4 P 9b64044f9b894140879afaec866e1092 [term 1 FOLLOWER]: Refusing update from remote peer d5ebccc255f349f5be67731ae451bf8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:27:45.725656 12145 consensus_queue.cc:1035] T af1b6ce82a1446da882212a070c823b4 P d5ebccc255f349f5be67731ae451bf8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b64044f9b894140879afaec866e1092" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:27:45.745054 12148 mvcc.cc:205] Tried to move back new op lower bound from 7334704192368504832 to 7334704192127188992. Current Snapshot: MvccSnapshot[applied={T|T < 7334704192368504832}]
I20260929 16:27:45.745532 12149 mvcc.cc:205] Tried to move back new op lower bound from 7334704192368504832 to 7334704192127188992. Current Snapshot: MvccSnapshot[applied={T|T < 7334704192368504832}]
I20260929 16:27:45.748222 12151 mvcc.cc:205] Tried to move back new op lower bound from 7334704192368504832 to 7334704192127188992. Current Snapshot: MvccSnapshot[applied={T|T < 7334704192368504832}]
I20260929 16:27:45.758972 11630 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:27:45.759184 11630 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:27:45.762403 11630 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1870ccd1ecf44112a65763c2051bc076: Sending DeleteTablet for 3 replicas of tablet af1b6ce82a1446da882212a070c823b4
I20260929 16:27:45.763792 11759 tablet_service.cc:1514] Processing DeleteTablet for tablet af1b6ce82a1446da882212a070c823b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:27:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40566
I20260929 16:27:45.764374 11914 tablet_service.cc:1514] Processing DeleteTablet for tablet af1b6ce82a1446da882212a070c823b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:27:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59542
I20260929 16:27:45.765887 12048 tablet_service.cc:1514] Processing DeleteTablet for tablet af1b6ce82a1446da882212a070c823b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:27:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45640
I20260929 16:27:45.767217 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 11711
I20260929 16:27:45.768198 12158 tablet_replica.cc:330] T af1b6ce82a1446da882212a070c823b4 P 9b64044f9b894140879afaec866e1092: stopping tablet replica
I20260929 16:27:45.768560 12158 raft_consensus.cc:2229] T af1b6ce82a1446da882212a070c823b4 P 9b64044f9b894140879afaec866e1092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:27:45.768934 12158 raft_consensus.cc:2258] T af1b6ce82a1446da882212a070c823b4 P 9b64044f9b894140879afaec866e1092 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 16:27:45.778369 11616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34865 (error 108)
I20260929 16:27:45.780050 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:27:45.779686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34865 (local address 127.0.0.1:40592)
0929 16:27:45.779872 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 16:27:45.779947 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260929 16:27:45.780385 11616 catalog_manager.cc:4584] TS d5ebccc255f349f5be67731ae451bf8b (127.0.98.1:34865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af1b6ce82a1446da882212a070c823b4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34865: connect: Connection refused (error 111)
I20260929 16:27:45.780788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 11850
I20260929 16:27:45.783646 12159 tablet_replica.cc:330] T af1b6ce82a1446da882212a070c823b4 P a324b52a8cac494380b9fb79fe2a8e01: stopping tablet replica
I20260929 16:27:45.784060 12159 raft_consensus.cc:2229] T af1b6ce82a1446da882212a070c823b4 P a324b52a8cac494380b9fb79fe2a8e01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:27:45.784464 12159 raft_consensus.cc:2258] T af1b6ce82a1446da882212a070c823b4 P a324b52a8cac494380b9fb79fe2a8e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:27:45.786195 12159 ts_tablet_manager.cc:1905] T af1b6ce82a1446da882212a070c823b4 P a324b52a8cac494380b9fb79fe2a8e01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:27:45.789285 12159 ts_tablet_manager.cc:1918] T af1b6ce82a1446da882212a070c823b4 P a324b52a8cac494380b9fb79fe2a8e01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:27:45.789477 12159 log.cc:1196] T af1b6ce82a1446da882212a070c823b4 P a324b52a8cac494380b9fb79fe2a8e01: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/af1b6ce82a1446da882212a070c823b4
I20260929 16:27:45.790283 12159 ts_tablet_manager.cc:1939] T af1b6ce82a1446da882212a070c823b4 P a324b52a8cac494380b9fb79fe2a8e01: Deleting consensus metadata
I20260929 16:27:45.792435 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:27:45.792096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33167 (local address 127.0.0.1:59550)
0929 16:27:45.792288 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 16:27:45.792344 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260929 16:27:45.792763 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 11985
W20260929 16:27:45.801944 12129 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:27:45.801361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44447 (local address 127.0.0.1:45648)
0929 16:27:45.801536 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 16:27:45.801550 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:27:45.801636 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:27:45.801821 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:44447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 16:27:45.802655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 11598
I20260929 16:27:45.808671 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 16:27:46.144326 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 16:27:46.144449 392 mini_ranger.cc:66] Stopping Ranger...
I20260929 16:27:46.489985 392 mini_ranger.cc:68] Stopped Ranger
2026-09-29 16:27:46.490 UTC [11184] LOG: received smart shutdown request
2026-09-29 16:27:46.494 UTC [11184] LOG: background worker "logical replication launcher" (PID 11193) exited with exit code 1
2026-09-29 16:27:46.494 UTC [11188] LOG: shutting down
2026-09-29 16:27:46.691 UTC [11184] LOG: database system is shut down
2026-09-29T16:27:46Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48686 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:27:46 dist-test-slave-vq9v krb5kdc[12169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:27:46 dist-test-slave-vq9v krb5kdc[12169](info): set up 2 sockets
Sep 29 16:27:46 dist-test-slave-vq9v krb5kdc[12169](info): commencing operation
krb5kdc: starting...
W20260929 16:27:47.889503 392 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:27:47 dist-test-slave-vq9v krb5kdc[12169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:27:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:27:47Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260929 16:27:48.068151 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task0hb3v3/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/postgres -l logfile start
2026-09-29 16:27:50.874 UTC [12199] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 16:27:50.874 UTC [12199] LOG: listening on IPv4 address "127.0.98.20", port 54359
2026-09-29 16:27:50.878 UTC [12199] LOG: listening on Unix socket "/tmp/.s.PGSQL.54359"
2026-09-29 16:27:50.913 UTC [12202] LOG: database system was shut down at 2026-09-29 16:27:49 UTC
2026-09-29 16:27:50.918 UTC [12199] LOG: database system is ready to accept connections
I20260929 16:27:51.865029 392 mini_postgres.cc:96] Postgres bound to 54359
2026-09-29 16:27:51.871 UTC [12210] FATAL: database "slave" does not exist
127.0.98.20:54359 - accepting connections
I20260929 16:27:51.871908 392 mini_ranger.cc:161] Starting Ranger...
I20260929 16:27:51.890527 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 16:27:52.465842 392 mini_ranger.cc:87] Created ranger Postgres database
I20260929 16:27:52.465938 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ranger-admin
2026-09-29 16:27:52,744 [I] DB FLAVOR :POSTGRES
2026-09-29 16:27:52,745 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:27:52,745 [I] Checking connection..
2026-09-29 16:27:52,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 16:27:53,011 [I] Checking connection passed.
2026-09-29 16:27:53,011 [I] --------- Verifying version history table ---------
2026-09-29 16:27:53,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:27:53,257 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:27:53,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:27:53,520 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:27:53,521 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 16:27:53,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 16:27:53.755 UTC [12274] WARNING: there is no transaction in progress
2026-09-29 16:27:53,761 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 16:27:53,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:27:54,030 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 16:27:54,030 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:27:54,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 16:27:54,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 16:27:54,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vq9v.c.gcp-upstream.internal','N') ;"
2026-09-29 16:27:54,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:27:55,013 [I] Table x_portal_user does not exist in database ranger
2026-09-29 16:27:55,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:27:55,293 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 16:27:55,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:27:55,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:27:55,806 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 16:27:55,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 16:27:56.987 UTC [12382] WARNING: there is no transaction in progress
2026-09-29 16:27:57.006 UTC [12382] WARNING: there is no transaction in progress
2026-09-29 16:27:57.029 UTC [12382] WARNING: there is no transaction in progress
2026-09-29 16:27:57.049 UTC [12382] WARNING: there is no transaction in progress
2026-09-29 16:27:57.067 UTC [12382] WARNING: there is no transaction in progress
2026-09-29 16:27:57.088 UTC [12382] WARNING: there is no transaction in progress
2026-09-29 16:27:57.155 UTC [12382] WARNING: there is no transaction in progress
2026-09-29 16:27:57.163 UTC [12382] WARNING: there is no transaction in progress
2026-09-29 16:27:57.181 UTC [12382] WARNING: there is no transaction in progress
2026-09-29 16:27:57.193 UTC [12382] WARNING: there is no transaction in progress
2026-09-29 16:27:57.207 UTC [12382] WARNING: there is no transaction in progress
2026-09-29 16:27:57.228 UTC [12382] WARNING: there is no transaction in progress
2026-09-29 16:27:57.241 UTC [12382] WARNING: there is no transaction in progress
2026-09-29 16:27:57.249 UTC [12382] WARNING: there is no transaction in progress
2026-09-29 16:27:57.257 UTC [12382] WARNING: there is no transaction in progress
2026-09-29 16:27:57.266 UTC [12382] WARNING: there is no transaction in progress
2026-09-29 16:27:58,120 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 16:27:58,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 16:27:58,367 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 16:27:58,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:27:58,605 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 16:27:58,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:27:58,867 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 16:27:58,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:27:59,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:27:59,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vq9v.c.gcp-upstream.internal';"
2026-09-29 16:27:59,618 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 16:27:59,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 16:27:59,890 [I] DB_PATCHES have already been applied
I20260929 16:27:59.895846 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task0hb3v3/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 16:27:59.895910 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260929 16:27:59.899432 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:42849
Sep 29, 2026 4:28:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:28:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:28:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 4:28:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 4:28:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 4:28:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 4:28:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 16:28:01.058761 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.187s user 0.002s sys 0.008s
Sep 29, 2026 4:28:22 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 4:28:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 4:28:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 4:28:22 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 4:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 4:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 4:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 4:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 4:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 4:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 4:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 4:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 4:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 4:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 4:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 4:28:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 4:28:22 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 4:28:23 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 16:28:24.089423 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 16:28:24 dist-test-slave-vq9v krb5kdc[12169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699304, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 16:28:24.181353 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 16:28:24.191370 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 16:28:25.219770 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 16:28:25.219856 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ranger-kms
I20260929 16:28:25.219879 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:54359
2026-09-29 16:28:25,388 [I] DB FLAVOR :POSTGRES
2026-09-29 16:28:25,389 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:28:25,389 [I] Checking connection
2026-09-29 16:28:25,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 16:28:25,647 [I] connection success
2026-09-29 16:28:25,647 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 16:28:25,647 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 16:28:25,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 16:28:25,901 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 16:28:25,901 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:28:25,901 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 16:28:25,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54359/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 16:28:26,165 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 16:28:26.684713 392 mini_ranger_kms.cc:310] Created kms service
I20260929 16:28:26.898065 392 mini_ranger_kms.cc:326] Created kudu user
I20260929 16:28:26.942320 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 16:28:26 dist-test-slave-vq9v krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699304, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 16:28:27.393244 392 mini_ranger_kms.cc:384] Added ranger policy
I20260929 16:28:27.393414 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task0hb3v3/build/debug/bin/postgresql.jar:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task0hb3v3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 16:28:27.393437 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260929 16:28:27.397817 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12563
I20260929 16:28:27.397934 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:52577
Sep 29, 2026 4:28:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:28:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:28:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 4:28:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 4:28:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 4:28:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 16:28:29.664489 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.483s user 0.009s sys 0.143s
I20260929 16:28:29.664633 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 16:28:29.664740 392 mini_ranger_kms.cc:401] 127.0.98.20:52577/kms/v1/keys
Sep 29 16:28:29 dist-test-slave-vq9v krb5kdc[12169](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790699309, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 16:28:30 dist-test-slave-vq9v krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699309, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 16:28:30 dist-test-slave-vq9v krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699309, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 16:28:32 dist-test-slave-vq9v krb5kdc[12169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:28:32.877785 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36193
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36193
--ranger_config_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52577/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:32.991745 12613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:32.992067 12613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:32.992156 12613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:32.996313 12613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:28:32.996404 12613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:32.996443 12613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:32.996500 12613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:28:32.996537 12613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:28:33.002208 12613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:52577/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36193
--ranger_config_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:33.003355 12613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:33.004426 12613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:33.013509 12618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:33.013548 12619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:33.013757 12613 server_base.cc:1034] running on GCE node
W20260929 16:28:33.013521 12621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:33.014597 12613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:33.015734 12613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:33.016956 12613 hybrid_clock.cc:648] HybridClock initialized: now 1790699313016938 us; error 40 us; skew 500 ppm
Sep 29 16:28:33 dist-test-slave-vq9v krb5kdc[12169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:33.023751 12613 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:28:33.024852 12613 webserver.cc:460] Webserver started at http://127.0.98.62:40293/ using document root <none> and password file <none>
I20260929 16:28:33.025492 12613 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:33.025575 12613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:33.025806 12613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:28:33 dist-test-slave-vq9v krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 16:28:33.087487 12613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "502009154d5248d9b6a6b4d01cd550aa"
format_stamp: "Formatted at 2026-09-29 16:28:33 on dist-test-slave-vq9v"
server_key: "7ac94df1c15567da8f6f7406e9560115"
server_key_iv: "89bf146c7c198e236578ddd0799bbc68"
server_key_version: "kuduclusterkey@0"
I20260929 16:28:33.088179 12613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "502009154d5248d9b6a6b4d01cd550aa"
format_stamp: "Formatted at 2026-09-29 16:28:33 on dist-test-slave-vq9v"
server_key: "7ac94df1c15567da8f6f7406e9560115"
server_key_iv: "89bf146c7c198e236578ddd0799bbc68"
server_key_version: "kuduclusterkey@0"
I20260929 16:28:33.092408 12613 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 16:28:33.106988 12628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:33.108457 12613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:28:33.108573 12613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "502009154d5248d9b6a6b4d01cd550aa"
format_stamp: "Formatted at 2026-09-29 16:28:33 on dist-test-slave-vq9v"
server_key: "7ac94df1c15567da8f6f7406e9560115"
server_key_iv: "89bf146c7c198e236578ddd0799bbc68"
server_key_version: "kuduclusterkey@0"
I20260929 16:28:33.108737 12613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:33.138346 12613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:33.141752 12613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:33.141969 12613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:33.150538 12680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36193 every 8 connection(s)
I20260929 16:28:33.150538 12613 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36193
I20260929 16:28:33.151880 12613 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:28:33.155059 12681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:33.160187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 12613
I20260929 16:28:33.160349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:28:33.160655 392 external_mini_cluster.cc:1428] Setting key 50e367dbeb7f4df0a5455e2cc37c2b3f
I20260929 16:28:33.161137 12681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa: Bootstrap starting.
I20260929 16:28:33.163674 12681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:33.164518 12681 log.cc:824] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:33.167009 12681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa: No bootstrap required, opened a new log
Sep 29 16:28:33 dist-test-slave-vq9v krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:28:33.171424 12681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502009154d5248d9b6a6b4d01cd550aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36193 } }
I20260929 16:28:33.171761 12681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:33.171873 12681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502009154d5248d9b6a6b4d01cd550aa, State: Initialized, Role: FOLLOWER
I20260929 16:28:33.172418 12681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502009154d5248d9b6a6b4d01cd550aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36193 } }
I20260929 16:28:33.172627 12681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:28:33.172708 12681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:28:33.172868 12681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:33.174026 12681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502009154d5248d9b6a6b4d01cd550aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36193 } }
I20260929 16:28:33.174448 12681 leader_election.cc:302] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 502009154d5248d9b6a6b4d01cd550aa; no voters:
I20260929 16:28:33.174860 12681 leader_election.cc:288] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:28:33.174988 12686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:28:33.175266 12686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [term 1 LEADER]: Becoming Leader. State: Replica: 502009154d5248d9b6a6b4d01cd550aa, State: Running, Role: LEADER
I20260929 16:28:33.175643 12686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502009154d5248d9b6a6b4d01cd550aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36193 } }
I20260929 16:28:33.176370 12681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:28:33.177018 12686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "502009154d5248d9b6a6b4d01cd550aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502009154d5248d9b6a6b4d01cd550aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36193 } } }
I20260929 16:28:33.177238 12686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:33.177486 12686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 502009154d5248d9b6a6b4d01cd550aa. Latest consensus state: current_term: 1 leader_uuid: "502009154d5248d9b6a6b4d01cd550aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502009154d5248d9b6a6b4d01cd550aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36193 } } }
I20260929 16:28:33.177608 12686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:33.178439 12684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:33.162140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45380 (local address 127.0.98.62:36193)
0929 16:28:33.162682 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:33.162702 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:33.162746 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:28:33.163582 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:33.163607 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:33.164037 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:33.164414 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:33.165354 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:33.166768 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:33.167982 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:33.168751 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:33.168979 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:33.172140 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:33.172147 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:33.172168 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:33.172206 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:33.174335 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:33.174907 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:33.174915 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:33.174923 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:33.175058 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:33.175385 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:33.175388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:33.175390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:33.175914 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:33.176119 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:33.177036 (+ 917us) server_negotiation.cc:299] Negotiation successful
0929 16:28:33.177392 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":188,"threads_started":1}
I20260929 16:28:33.178715 12681 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 16:28:33.181545 12693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:28:33.187608 12693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:28:33.196103 12693 catalog_manager.cc:1347] Generated new cluster ID: 581fd826efb14dd18c3708f715e72b14
I20260929 16:28:33.196170 12693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:28:33.216799 12693 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:28:33.218220 12693 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:28:33.225073 12693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa: Generated new TSK 0
I20260929 16:28:33.226085 12693 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 16:28:33.938232 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52577/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36193
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:34.081318 12726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:34.081612 12726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:34.081657 12726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:34.086786 12726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:34.086886 12726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:34.086992 12726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:28:34.094547 12726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:52577/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:34.095795 12726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:34.096872 12726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:34.108667 12733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:34.108906 12736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:34.108647 12734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:34.109447 12726 server_base.cc:1034] running on GCE node
I20260929 16:28:34.110010 12726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:34.110903 12726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:34.112092 12726 hybrid_clock.cc:648] HybridClock initialized: now 1790699314112070 us; error 35 us; skew 500 ppm
Sep 29 16:28:34 dist-test-slave-vq9v krb5kdc[12169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:34.115998 12726 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:28:34.117249 12726 webserver.cc:460] Webserver started at http://127.0.98.1:33331/ using document root <none> and password file <none>
I20260929 16:28:34.117957 12726 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:34.118031 12726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:34.118279 12726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:28:34 dist-test-slave-vq9v krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 16:28:34.136157 12726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "a446ebbc1d634d8796f6f1414b79a72a"
format_stamp: "Formatted at 2026-09-29 16:28:34 on dist-test-slave-vq9v"
server_key: "3b7abbde5c50584ed79618826fe4881e"
server_key_iv: "aea1b44cb87343a331f2ae0bb2da070e"
server_key_version: "kuduclusterkey@0"
I20260929 16:28:34.136940 12726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a446ebbc1d634d8796f6f1414b79a72a"
format_stamp: "Formatted at 2026-09-29 16:28:34 on dist-test-slave-vq9v"
server_key: "3b7abbde5c50584ed79618826fe4881e"
server_key_iv: "aea1b44cb87343a331f2ae0bb2da070e"
server_key_version: "kuduclusterkey@0"
I20260929 16:28:34.142201 12726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 16:28:34.168970 12743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:34.172353 12726 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.003s sys 0.001s
I20260929 16:28:34.172588 12726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "a446ebbc1d634d8796f6f1414b79a72a"
format_stamp: "Formatted at 2026-09-29 16:28:34 on dist-test-slave-vq9v"
server_key: "3b7abbde5c50584ed79618826fe4881e"
server_key_iv: "aea1b44cb87343a331f2ae0bb2da070e"
server_key_version: "kuduclusterkey@0"
I20260929 16:28:34.172847 12726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:34.208227 12726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:34.213533 12726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:34.213816 12726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:34.214762 12726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:28:34.216008 12726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:34.216116 12726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 16:28:34.216250 12726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:34.216337 12726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:34.229755 12726 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42573
I20260929 16:28:34.231171 12726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:28:34.229830 12858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42573 every 8 connection(s)
I20260929 16:28:34.233346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 12726
I20260929 16:28:34.233512 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:28:34.233852 392 external_mini_cluster.cc:1428] Setting key 115091f4767a7264fdbc32a845cea234
Sep 29 16:28:34 dist-test-slave-vq9v krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 16:28:34.255264 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:34.233953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60919 (local address 127.0.98.62:36193)
0929 16:28:34.234423 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:34.234428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:34.237135 (+ 2707us) server_negotiation.cc:407] Connection header received
0929 16:28:34.237185 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:34.237188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:34.237294 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:34.237394 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:34.239017 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:34.239835 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:34.241625 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:34.241984 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:34.242065 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:34.245589 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:34.245593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:34.245597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:34.245634 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:34.247678 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:34.248357 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:34.248361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:34.248362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:34.248466 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:34.248899 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:34.248903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:34.248905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:34.249154 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:34.249287 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:34.254888 (+ 5601us) server_negotiation.cc:299] Negotiation successful
0929 16:28:34.255049 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":75,"threads_started":1}
I20260929 16:28:34.257370 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:34.234494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36193 (local address 127.0.98.1:60919)
0929 16:28:34.235653 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:34.235712 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:34.236824 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:34.237743 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:34.237756 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:34.238130 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:34.238832 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:34.238853 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:34.239986 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:34.239989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:34.241478 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:34.241490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:34.242288 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:34.242293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:34.242501 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:34.243613 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:34.243636 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:34.245375 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:34.247822 (+ 2447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:34.247828 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:34.247846 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:34.248214 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:34.248586 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:34.248588 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:34.248591 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:34.248776 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:34.249375 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:34.249382 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:34.249750 (+ 368us) client_negotiation.cc:769] Sending connection context
0929 16:28:34.255548 (+ 5798us) client_negotiation.cc:240] Negotiation successful
0929 16:28:34.256152 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 16:28:34.260542 12859 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36193
I20260929 16:28:34.260812 12859 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:34.261446 12859 heartbeater.cc:511] Master 127.0.98.62:36193 requested a full tablet report, sending...
I20260929 16:28:34.263302 12645 ts_manager.cc:194] Registered new tserver with Master: a446ebbc1d634d8796f6f1414b79a72a (127.0.98.1:42573)
I20260929 16:28:34.264858 12645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 16:28:34.277909 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52577/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36193
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:28:34 dist-test-slave-vq9v krb5kdc[12169](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790699314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 16:28:34.430954 12865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:34.431378 12865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:34.431448 12865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:34.437779 12865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:34.437878 12865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:34.437973 12865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:28:34.444991 12865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:52577/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:34.446493 12865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:34.447893 12865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:34.460839 12872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:34.460995 12874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:34.460852 12871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:34.462298 12865 server_base.cc:1034] running on GCE node
I20260929 16:28:34.462826 12865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:34.463653 12865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:34.464857 12865 hybrid_clock.cc:648] HybridClock initialized: now 1790699314464834 us; error 38 us; skew 500 ppm
Sep 29 16:28:34 dist-test-slave-vq9v krb5kdc[12169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:34.468580 12865 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:28:34.469724 12865 webserver.cc:460] Webserver started at http://127.0.98.2:38717/ using document root <none> and password file <none>
I20260929 16:28:34.470420 12865 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:34.470491 12865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:34.470709 12865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:28:34 dist-test-slave-vq9v krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 16:28:34.498664 12865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "826654ec8c9a41fd97123de0216b44a5"
format_stamp: "Formatted at 2026-09-29 16:28:34 on dist-test-slave-vq9v"
server_key: "81eed241f4beaa09cb6c0baaadd7b477"
server_key_iv: "98b3a9f1ae20746eeef0fa14b94ba3e7"
server_key_version: "kuduclusterkey@0"
I20260929 16:28:34.499861 12865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "826654ec8c9a41fd97123de0216b44a5"
format_stamp: "Formatted at 2026-09-29 16:28:34 on dist-test-slave-vq9v"
server_key: "81eed241f4beaa09cb6c0baaadd7b477"
server_key_iv: "98b3a9f1ae20746eeef0fa14b94ba3e7"
server_key_version: "kuduclusterkey@0"
I20260929 16:28:34.505491 12865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 16:28:34.533622 12881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:34.535502 12865 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260929 16:28:34.535658 12865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "826654ec8c9a41fd97123de0216b44a5"
format_stamp: "Formatted at 2026-09-29 16:28:34 on dist-test-slave-vq9v"
server_key: "81eed241f4beaa09cb6c0baaadd7b477"
server_key_iv: "98b3a9f1ae20746eeef0fa14b94ba3e7"
server_key_version: "kuduclusterkey@0"
I20260929 16:28:34.535828 12865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:34.575130 12865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:34.579134 12865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:34.579401 12865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:34.580116 12865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:28:34.581092 12865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:34.581162 12865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:34.581254 12865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:34.581290 12865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:34.592468 12865 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46441
I20260929 16:28:34.592490 12994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46441 every 8 connection(s)
I20260929 16:28:34.593932 12865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:28:34.598104 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 12865
I20260929 16:28:34.598232 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:28:34.598579 392 external_mini_cluster.cc:1428] Setting key abc4f86bde948023e146218087fd9e5d
Sep 29 16:28:34 dist-test-slave-vq9v krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:28:34.613758 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:34.598448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53871 (local address 127.0.98.62:36193)
0929 16:28:34.598616 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:34.598622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:34.599005 (+ 383us) server_negotiation.cc:407] Connection header received
0929 16:28:34.600302 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:34.600305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:34.600356 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:34.600408 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:34.602284 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:34.603147 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:34.605048 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:34.605438 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:34.605524 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:34.608911 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:34.608915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:34.608918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:34.608956 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:34.610874 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:34.611641 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:34.611646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:34.611648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:34.611754 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:34.612182 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:34.612186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:34.612187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:34.612421 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:34.612493 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:34.613377 (+ 884us) server_negotiation.cc:299] Negotiation successful
0929 16:28:34.613557 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 16:28:34.615799 12997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:34.597247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36193 (local address 127.0.98.2:53871)
0929 16:28:34.598823 (+ 1576us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:34.598885 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:34.600051 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:34.600853 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:34.600865 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:34.601291 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:34.602092 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:34.602114 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:34.603333 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:34.603339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:34.604878 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:34.604894 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:34.605767 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:34.605771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:34.605983 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:34.606775 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:34.606802 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:34.608649 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:34.611041 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:34.611047 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:34.611067 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:34.611459 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:34.611876 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:34.611879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:34.611882 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:34.612070 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:34.612574 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:34.612582 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:34.613044 (+ 462us) client_negotiation.cc:769] Sending connection context
0929 16:28:34.614253 (+ 1209us) client_negotiation.cc:240] Negotiation successful
0929 16:28:34.614646 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1321,"thread_start_us":162,"threads_started":1}
I20260929 16:28:34.617326 12995 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36193
I20260929 16:28:34.617606 12995 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 16:28:34.618239 12995 heartbeater.cc:511] Master 127.0.98.62:36193 requested a full tablet report, sending...
I20260929 16:28:34.619443 12645 ts_manager.cc:194] Registered new tserver with Master: 826654ec8c9a41fd97123de0216b44a5 (127.0.98.2:46441)
I20260929 16:28:34.620216 12645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 16:28:34.640981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52577/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36193
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:28:34 dist-test-slave-vq9v krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260929 16:28:34.814613 13000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:34.815019 13000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:34.815096 13000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:34.820932 13000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:34.821043 13000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:34.821152 13000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:28:34.828900 13000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:52577/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:34.830566 13000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:34.832278 13000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:34.846095 13008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:34.846124 13006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:34.846133 13005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:34.846591 13000 server_base.cc:1034] running on GCE node
I20260929 16:28:34.847071 13000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:34.847805 13000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:34.849021 13000 hybrid_clock.cc:648] HybridClock initialized: now 1790699314848983 us; error 55 us; skew 500 ppm
Sep 29 16:28:34 dist-test-slave-vq9v krb5kdc[12169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:34.852088 13000 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:28:34.853083 13000 webserver.cc:460] Webserver started at http://127.0.98.3:46681/ using document root <none> and password file <none>
I20260929 16:28:34.853714 13000 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:34.853786 13000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:34.854014 13000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:28:34 dist-test-slave-vq9v krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 16:28:34.870664 13000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "676038e975bb4f3b88fd4b74fd06a7f9"
format_stamp: "Formatted at 2026-09-29 16:28:34 on dist-test-slave-vq9v"
server_key: "7728d97e73721c031325df107f889725"
server_key_iv: "9143819811dd4075c662e0af1ef58d04"
server_key_version: "kuduclusterkey@0"
I20260929 16:28:34.871528 13000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "676038e975bb4f3b88fd4b74fd06a7f9"
format_stamp: "Formatted at 2026-09-29 16:28:34 on dist-test-slave-vq9v"
server_key: "7728d97e73721c031325df107f889725"
server_key_iv: "9143819811dd4075c662e0af1ef58d04"
server_key_version: "kuduclusterkey@0"
I20260929 16:28:34.890959 13000 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.004s sys 0.000s
I20260929 16:28:34.908422 13015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:34.910138 13000 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:28:34.910291 13000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "676038e975bb4f3b88fd4b74fd06a7f9"
format_stamp: "Formatted at 2026-09-29 16:28:34 on dist-test-slave-vq9v"
server_key: "7728d97e73721c031325df107f889725"
server_key_iv: "9143819811dd4075c662e0af1ef58d04"
server_key_version: "kuduclusterkey@0"
I20260929 16:28:34.910445 13000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:34.931900 13000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:34.935957 13000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:34.936170 13000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:34.936914 13000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:28:34.938073 13000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:34.938143 13000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:34.938199 13000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:34.938233 13000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:34.949659 13000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36321
I20260929 16:28:34.950035 13128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36321 every 8 connection(s)
I20260929 16:28:34.951139 13000 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:28:34.953658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 13000
I20260929 16:28:34.953780 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:28:34.954092 392 external_mini_cluster.cc:1428] Setting key 5d02f35459583629390ff53a55a2bd0f
Sep 29 16:28:34 dist-test-slave-vq9v krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:28:34.971757 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:34.954475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36569 (local address 127.0.98.62:36193)
0929 16:28:34.954894 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:34.954899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:34.955950 (+ 1051us) server_negotiation.cc:407] Connection header received
0929 16:28:34.958030 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:34.958034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:34.958088 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:34.958139 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:34.959934 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:34.960768 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:34.962625 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:34.962958 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:34.963045 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:34.966470 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:34.966475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:34.966478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:34.966509 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:34.968813 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:34.969736 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:34.969740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:34.969742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:34.969840 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:34.970320 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:34.970324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:34.970326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:34.970578 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:34.970677 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:34.971453 (+ 776us) server_negotiation.cc:299] Negotiation successful
0929 16:28:34.971587 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319}
I20260929 16:28:34.972900 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:34.954139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36193 (local address 127.0.98.3:36569)
0929 16:28:34.955781 (+ 1642us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:34.955844 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:34.957754 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:34.958591 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:34.958601 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:34.958994 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:34.959733 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:34.959754 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:34.960954 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:34.960957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:34.962459 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:34.962472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:34.963283 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:34.963286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:34.963523 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:34.964322 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:34.964344 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:34.966208 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:34.968960 (+ 2752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:34.968968 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:34.968988 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:34.969611 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:34.969965 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:34.969968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:34.969970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:34.970185 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:34.970710 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:34.970719 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:34.971115 (+ 396us) client_negotiation.cc:769] Sending connection context
0929 16:28:34.971349 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 16:28:34.971724 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"thread_start_us":158,"threads_started":1}
I20260929 16:28:34.974239 13129 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36193
I20260929 16:28:34.974534 13129 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:34.975124 13129 heartbeater.cc:511] Master 127.0.98.62:36193 requested a full tablet report, sending...
I20260929 16:28:34.976339 12645 ts_manager.cc:194] Registered new tserver with Master: 676038e975bb4f3b88fd4b74fd06a7f9 (127.0.98.3:36321)
I20260929 16:28:34.977087 12645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36569
I20260929 16:28:34.981881 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:28:34.982689 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 12726
I20260929 16:28:34.990950 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 12865
I20260929 16:28:34.996821 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 13000
I20260929 16:28:35.003940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 12613
I20260929 16:28:35.012506 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36193
--webserver_interface=127.0.98.62
--webserver_port=40293
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36193
--ranger_config_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52577/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52577/kms with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:35.130367 13132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:35.130667 13132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:35.130709 13132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:35.135125 13132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:28:35.135203 13132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:35.135223 13132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:35.135241 13132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:28:35.135258 13132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:28:35.141309 13132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52577/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36193
--ranger_config_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:35.142385 13132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:35.143503 13132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:35.153546 13140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:35.153616 13132 server_base.cc:1034] running on GCE node
W20260929 16:28:35.153594 13138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:35.153549 13137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:35.154541 13132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:35.155694 13132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:35.156857 13132 hybrid_clock.cc:648] HybridClock initialized: now 1790699315156847 us; error 34 us; skew 500 ppm
Sep 29 16:28:35 dist-test-slave-vq9v krb5kdc[12169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:35.159932 13132 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:28:35.160913 13132 webserver.cc:460] Webserver started at http://127.0.98.62:40293/ using document root <none> and password file <none>
I20260929 16:28:35.161576 13132 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:35.161688 13132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:28:35 dist-test-slave-vq9v krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 16:28:35.210309 13132 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:28:35.212642 13148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:35.214118 13132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:28:35.214265 13132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "502009154d5248d9b6a6b4d01cd550aa"
format_stamp: "Formatted at 2026-09-29 16:28:33 on dist-test-slave-vq9v"
server_key: "7ac94df1c15567da8f6f7406e9560115"
server_key_iv: "89bf146c7c198e236578ddd0799bbc68"
server_key_version: "kuduclusterkey@0"
I20260929 16:28:35.214884 13132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:35.229898 13132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:35.233268 13132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:35.233479 13132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:35.240789 13132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36193
I20260929 16:28:35.240833 13200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36193 every 8 connection(s)
I20260929 16:28:35.241902 13132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:28:35.246145 13201 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:28:35.247759 13201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa: Bootstrap starting.
I20260929 16:28:35.250708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 13132
I20260929 16:28:35.251186 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52577/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42573
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36193
--webserver_port=33331
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52577/kms with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:28:35.265115 13201 log.cc:824] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:35.271881 13201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:28:35.272337 13201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa: Bootstrap complete.
I20260929 16:28:35.276613 13201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502009154d5248d9b6a6b4d01cd550aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36193 } }
I20260929 16:28:35.276999 13201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502009154d5248d9b6a6b4d01cd550aa, State: Initialized, Role: FOLLOWER
I20260929 16:28:35.277520 13201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502009154d5248d9b6a6b4d01cd550aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36193 } }
I20260929 16:28:35.277632 13201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:28:35.277704 13201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:28:35.277801 13201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:28:35.279747 13201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502009154d5248d9b6a6b4d01cd550aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36193 } }
I20260929 16:28:35.280092 13201 leader_election.cc:302] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 502009154d5248d9b6a6b4d01cd550aa; no voters:
I20260929 16:28:35.280454 13201 leader_election.cc:288] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:28:35.280544 13205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:28:35.280755 13205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [term 2 LEADER]: Becoming Leader. State: Replica: 502009154d5248d9b6a6b4d01cd550aa, State: Running, Role: LEADER
I20260929 16:28:35.281056 13205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502009154d5248d9b6a6b4d01cd550aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36193 } }
I20260929 16:28:35.281744 13201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:28:35.282063 13207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 502009154d5248d9b6a6b4d01cd550aa. Latest consensus state: current_term: 2 leader_uuid: "502009154d5248d9b6a6b4d01cd550aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502009154d5248d9b6a6b4d01cd550aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36193 } } }
I20260929 16:28:35.282220 13207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:35.282423 13206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "502009154d5248d9b6a6b4d01cd550aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502009154d5248d9b6a6b4d01cd550aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36193 } } }
I20260929 16:28:35.282526 13206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:35.283025 13209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:28:35.286461 13209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:28:35.287110 13201 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 16:28:35.287106 13209 catalog_manager.cc:1259] Loaded cluster ID: 581fd826efb14dd18c3708f715e72b14
I20260929 16:28:35.288093 13209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:28:35.294142 13209 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:28:35.295179 13209 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa: Loaded TSK: 0
I20260929 16:28:35.296669 13209 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:28:35.396124 13203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:35.396456 13203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:35.396530 13203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:35.402083 13203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:35.402176 13203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:35.402280 13203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:28:35.410368 13203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52577/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33331
--tserver_master_addrs=127.0.98.62:36193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:35.411608 13203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:35.412756 13203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:35.424896 13238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:35.424897 13241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:35.424902 13239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:35.425906 13203 server_base.cc:1034] running on GCE node
I20260929 16:28:35.426415 13203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:35.427235 13203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:35.430567 13203 hybrid_clock.cc:648] HybridClock initialized: now 1790699315430506 us; error 71 us; skew 500 ppm
Sep 29 16:28:35 dist-test-slave-vq9v krb5kdc[12169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:35.434346 13203 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:28:35.435441 13203 webserver.cc:460] Webserver started at http://127.0.98.1:33331/ using document root <none> and password file <none>
I20260929 16:28:35.436025 13203 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:35.436138 13203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:28:35 dist-test-slave-vq9v krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 16:28:35.461517 13203 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 16:28:35.474097 13249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:35.475885 13203 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.000s
I20260929 16:28:35.476040 13203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "a446ebbc1d634d8796f6f1414b79a72a"
format_stamp: "Formatted at 2026-09-29 16:28:34 on dist-test-slave-vq9v"
server_key: "3b7abbde5c50584ed79618826fe4881e"
server_key_iv: "aea1b44cb87343a331f2ae0bb2da070e"
server_key_version: "kuduclusterkey@0"
I20260929 16:28:35.476817 13203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:35.497359 13203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:35.501238 13203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:35.501467 13203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:35.502250 13203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:28:35.503360 13203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:35.503429 13203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:35.503494 13203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:35.503535 13203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:35.518158 13203 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42573
I20260929 16:28:35.519570 13203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:28:35.520510 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 13203
I20260929 16:28:35.521019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52577/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46441
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36193
--webserver_port=38717
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52577/kms with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:28:35.520939 13362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42573 every 8 connection(s)
Sep 29 16:28:35 dist-test-slave-vq9v krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:28:35.544694 13367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:35.523586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36193 (local address 127.0.98.1:39131)
0929 16:28:35.524234 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:35.524298 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:35.525442 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:35.527373 (+ 1931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:35.527384 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:35.527764 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:35.528518 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:35.528541 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:35.530937 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:35.530941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:35.532103 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:35.532114 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:35.534669 (+ 2555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:35.534677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:35.534955 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:35.535753 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:35.535782 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:35.537695 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:35.540686 (+ 2991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:35.540696 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:35.540717 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:35.541160 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:35.541705 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:35.541710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:35.541714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:35.541937 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:35.542718 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:35.542736 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:35.543054 (+ 318us) client_negotiation.cc:769] Sending connection context
0929 16:28:35.543234 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 16:28:35.543576 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":153,"threads_started":1}
I20260929 16:28:35.546571 13366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:35.523518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39131 (local address 127.0.98.62:36193)
0929 16:28:35.525989 (+ 2471us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:35.526012 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:35.526057 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:28:35.526375 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:35.526392 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:35.526836 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:35.528801 (+ 1965us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:35.529126 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:35.530753 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:35.533360 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:35.534431 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:35.535450 (+ 1019us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:35.538013 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:35.538022 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:35.538046 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:35.538090 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:35.540510 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:35.541378 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:35.541388 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:35.541396 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:35.541566 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:35.542073 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:35.542076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:35.542078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:35.542582 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:35.545143 (+ 2561us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:35.545357 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 16:28:35.545656 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2291,"thread_start_us":123,"threads_started":1}
I20260929 16:28:35.555223 13363 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36193
I20260929 16:28:35.555545 13363 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:35.556216 13363 heartbeater.cc:511] Master 127.0.98.62:36193 requested a full tablet report, sending...
W20260929 16:28:35.558632 13363 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36193 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:28:35.558777 13363 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:35.558964 13363 heartbeater.cc:511] Master 127.0.98.62:36193 requested a full tablet report, sending...
I20260929 16:28:35.559757 13363 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:35.559933 13363 heartbeater.cc:511] Master 127.0.98.62:36193 requested a full tablet report, sending...
W20260929 16:28:35.560619 13363 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:28:35.695130 13365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:35.695549 13365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:35.695633 13365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:35.700830 13365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:35.700937 13365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:35.701042 13365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:28:35.708827 13365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52577/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38717
--tserver_master_addrs=127.0.98.62:36193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:35.710287 13365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:35.711769 13365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:35.725001 13375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:35.725488 13373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:35.725610 13372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:35.726573 13365 server_base.cc:1034] running on GCE node
I20260929 16:28:35.727052 13365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:35.727945 13365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:35.731429 13365 hybrid_clock.cc:648] HybridClock initialized: now 1790699315731336 us; error 104 us; skew 500 ppm
Sep 29 16:28:35 dist-test-slave-vq9v krb5kdc[12169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:35.735029 13365 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:28:35.736255 13365 webserver.cc:460] Webserver started at http://127.0.98.2:38717/ using document root <none> and password file <none>
I20260929 16:28:35.736948 13365 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:35.737061 13365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:28:35 dist-test-slave-vq9v krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 16:28:35.756209 13365 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:28:35.759172 13383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:35.760864 13365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:28:35.761080 13365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "826654ec8c9a41fd97123de0216b44a5"
format_stamp: "Formatted at 2026-09-29 16:28:34 on dist-test-slave-vq9v"
server_key: "81eed241f4beaa09cb6c0baaadd7b477"
server_key_iv: "98b3a9f1ae20746eeef0fa14b94ba3e7"
server_key_version: "kuduclusterkey@0"
I20260929 16:28:35.761969 13365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:35.796478 13365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:35.800319 13365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:35.800598 13365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:35.801410 13365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:28:35.802469 13365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:35.802575 13365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:35.802671 13365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:35.802748 13365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:35.817049 13365 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46441
I20260929 16:28:35.818379 13365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:28:35.827333 13496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46441 every 8 connection(s)
I20260929 16:28:35.828554 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 13365
I20260929 16:28:35.829051 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52577/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36321
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36193
--webserver_port=46681
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52577/kms with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:28:35 dist-test-slave-vq9v krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:28:35.842473 13366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:35.820448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58703 (local address 127.0.98.62:36193)
0929 16:28:35.820581 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:35.820585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:35.822468 (+ 1883us) server_negotiation.cc:407] Connection header received
0929 16:28:35.823733 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:35.823738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:35.823809 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:35.823871 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:35.825595 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:35.826422 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:35.828050 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:35.828391 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:35.828502 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:35.837477 (+ 8975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:35.837484 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:35.837488 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:35.837545 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:35.839582 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:35.840447 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:35.840454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:35.840459 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:35.840559 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:35.841064 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:35.841067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:35.841069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:35.841334 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:35.841419 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:35.842155 (+ 736us) server_negotiation.cc:299] Negotiation successful
0929 16:28:35.842300 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:28:35.844208 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:35.821011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36193 (local address 127.0.98.2:58703)
0929 16:28:35.822298 (+ 1287us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:35.822343 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:35.823428 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:35.824297 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:35.824306 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:35.824667 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:35.825406 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:35.825428 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:35.826597 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:35.826601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:35.827898 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:35.827908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:35.828755 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:35.828759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:35.828992 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:35.830020 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:35.830048 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:35.832180 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:35.839792 (+ 7612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:35.839804 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:35.839830 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:35.840301 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:35.840689 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:35.840695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:35.840698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:35.840939 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:35.841487 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:35.841494 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:35.841873 (+ 379us) client_negotiation.cc:769] Sending connection context
0929 16:28:35.842671 (+ 798us) client_negotiation.cc:240] Negotiation successful
0929 16:28:35.843051 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"thread_start_us":217,"threads_started":1}
I20260929 16:28:35.845556 13497 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36193
I20260929 16:28:35.845806 13497 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:35.846335 13497 heartbeater.cc:511] Master 127.0.98.62:36193 requested a full tablet report, sending...
W20260929 16:28:35.847927 13497 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36193 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:28:35.848065 13497 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:35.848250 13497 heartbeater.cc:511] Master 127.0.98.62:36193 requested a full tablet report, sending...
I20260929 16:28:35.849095 13497 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:35.849325 13497 heartbeater.cc:511] Master 127.0.98.62:36193 requested a full tablet report, sending...
W20260929 16:28:35.850128 13497 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:28:35.990926 13500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:35.991365 13500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:35.991464 13500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:35.998430 13500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:35.998535 13500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:35.998636 13500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:28:36.007210 13500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52577/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46681
--tserver_master_addrs=127.0.98.62:36193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:36.008735 13500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:36.010213 13500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:36.024197 13507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:36.024410 13506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:36.024645 13509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:36.025077 13500 server_base.cc:1034] running on GCE node
I20260929 16:28:36.025797 13500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:36.026688 13500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:36.028003 13500 hybrid_clock.cc:648] HybridClock initialized: now 1790699316027916 us; error 101 us; skew 500 ppm
Sep 29 16:28:36 dist-test-slave-vq9v krb5kdc[12169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:36.031966 13500 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:28:36.033226 13500 webserver.cc:460] Webserver started at http://127.0.98.3:46681/ using document root <none> and password file <none>
I20260929 16:28:36.034047 13500 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:36.034178 13500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:28:36 dist-test-slave-vq9v krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 16:28:36.056779 13500 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 16:28:36.060007 13517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:36.061688 13500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:28:36.061851 13500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "676038e975bb4f3b88fd4b74fd06a7f9"
format_stamp: "Formatted at 2026-09-29 16:28:34 on dist-test-slave-vq9v"
server_key: "7728d97e73721c031325df107f889725"
server_key_iv: "9143819811dd4075c662e0af1ef58d04"
server_key_version: "kuduclusterkey@0"
I20260929 16:28:36.062618 13500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:36.098291 13500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:36.101841 13500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:36.102068 13500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:36.102677 13500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:28:36.103611 13500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:36.103681 13500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:36.103762 13500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:36.103798 13500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:36.115497 13500 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36321
I20260929 16:28:36.116143 13630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36321 every 8 connection(s)
I20260929 16:28:36.116874 13500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:28:36.118472 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 13500
Sep 29 16:28:36 dist-test-slave-vq9v krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:28:36.136931 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:36.120941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56816 (local address 127.0.98.62:36193)
0929 16:28:36.122255 (+ 1314us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:36.122259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:36.122272 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:28:36.122305 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:36.122311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:36.122351 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:36.122443 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:36.123501 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:36.124284 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:36.126165 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:36.126695 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:36.126806 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:36.129017 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:36.129021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:36.129025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:36.129047 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:36.131569 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:36.135152 (+ 3583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:36.135156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:36.135158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:36.135284 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:36.135626 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:36.135631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:36.135633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:36.135902 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:36.136134 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:36.136532 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 16:28:36.136683 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1244,"thread_start_us":89,"threads_started":1}
I20260929 16:28:36.137188 13366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:36.119346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45263 (local address 127.0.98.62:36193)
0929 16:28:36.121166 (+ 1820us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:36.121171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:36.121188 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:28:36.122056 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:36.122060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:36.122117 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:36.122207 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:36.123568 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:36.124448 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:36.126132 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:36.126448 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:36.126596 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:36.130621 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:36.130625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:36.130628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:36.130696 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:36.134172 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:36.135179 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:36.135183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:36.135185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:36.135315 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:36.135786 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:36.135790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:36.135793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:36.136016 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:36.136103 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:36.136917 (+ 814us) server_negotiation.cc:299] Negotiation successful
0929 16:28:36.137057 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":1747}
I20260929 16:28:36.139804 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:36.119869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36193 (local address 127.0.98.3:45263)
0929 16:28:36.120792 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:36.120851 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:36.121818 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:36.122552 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:36.122562 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:36.122862 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:36.123382 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:36.123396 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:36.124592 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:36.124596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:36.125993 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:36.126003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:36.126624 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:36.126629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:36.126913 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:36.127868 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:36.127896 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:36.130397 (+ 2501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:36.134396 (+ 3999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:36.134405 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:36.134427 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:36.135033 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:36.135430 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:36.135434 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:36.135436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:36.135647 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:36.136181 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:36.136188 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:36.136592 (+ 404us) client_negotiation.cc:769] Sending connection context
0929 16:28:36.138063 (+ 1471us) client_negotiation.cc:240] Negotiation successful
0929 16:28:36.138844 (+ 781us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":149,"threads_started":1}
I20260929 16:28:36.141310 13631 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36193
I20260929 16:28:36.141601 13631 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:36.142153 13631 heartbeater.cc:511] Master 127.0.98.62:36193 requested a full tablet report, sending...
W20260929 16:28:36.143829 13631 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36193 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:28:36.143973 13631 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:36.144197 13631 heartbeater.cc:511] Master 127.0.98.62:36193 requested a full tablet report, sending...
I20260929 16:28:36.145061 13631 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:36.145285 13631 heartbeater.cc:511] Master 127.0.98.62:36193 requested a full tablet report, sending...
W20260929 16:28:36.146077 13631 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 16:28:36 dist-test-slave-vq9v krb5kdc[12169](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790699316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:36.560873 13363 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:36.561282 13363 heartbeater.cc:511] Master 127.0.98.62:36193 requested a full tablet report, sending...
I20260929 16:28:36.562609 13165 ts_manager.cc:194] Registered new tserver with Master: a446ebbc1d634d8796f6f1414b79a72a (127.0.98.1:42573)
I20260929 16:28:36.564247 13165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39131
I20260929 16:28:36.850365 13497 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:36.850711 13497 heartbeater.cc:511] Master 127.0.98.62:36193 requested a full tablet report, sending...
I20260929 16:28:36.851665 13164 ts_manager.cc:194] Registered new tserver with Master: 826654ec8c9a41fd97123de0216b44a5 (127.0.98.2:46441)
I20260929 16:28:36.852538 13164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58703
Sep 29 16:28:36 dist-test-slave-vq9v krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 16:28:37.146368 13631 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:37.146730 13631 heartbeater.cc:511] Master 127.0.98.62:36193 requested a full tablet report, sending...
I20260929 16:28:37.147775 13164 ts_manager.cc:194] Registered new tserver with Master: 676038e975bb4f3b88fd4b74fd06a7f9 (127.0.98.3:36321)
I20260929 16:28:37.148674 13164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45263
I20260929 16:28:37.151865 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:28:37.165966 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:37.153700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56832 (local address 127.0.98.62:36193)
0929 16:28:37.155517 (+ 1817us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:37.155522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:37.155537 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:28:37.155581 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:37.155584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:37.155631 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:37.155730 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:37.156565 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.157397 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.158631 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.158958 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.159048 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:37.160856 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:37.160860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:37.160863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:37.160893 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:37.163879 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:37.164421 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:37.164425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:37.164427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:37.164527 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:37.164861 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:37.164865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:37.164870 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:37.165068 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:37.165237 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:37.165493 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 16:28:37.165630 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1736,"thread_start_us":76,"threads_started":1}
I20260929 16:28:37.170743 13164 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:28:37.174636 13164 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:28:37.203536 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:37.193415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46441 (local address 127.0.0.1:49052)
0929 16:28:37.195939 (+ 2524us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:37.195954 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:37.196108 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:37.196788 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:37.196793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:37.196811 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:37.196916 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:37.196920 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.199279 (+ 2359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.199282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:37.201535 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:37.201545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.202789 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.202792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:37.202917 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:37.202955 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 16:28:37.203064 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 16:28:37.203323 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2429,"thread_start_us":81,"threads_started":1}
I20260929 16:28:37.204941 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:37.194224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49052 (local address 127.0.98.2:46441)
0929 16:28:37.195213 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:37.195222 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:37.196035 (+ 813us) server_negotiation.cc:407] Connection header received
0929 16:28:37.196268 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:37.196278 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:37.196522 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:37.196621 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:37.197035 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.199142 (+ 2107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.201666 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.202482 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.202576 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:37.202725 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:37.203837 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0929 16:28:37.204777 (+ 940us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":132,"threads_started":1}
I20260929 16:28:37.206418 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:37.193316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36321 (local address 127.0.0.1:58788)
0929 16:28:37.195498 (+ 2182us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:37.195559 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:37.195691 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:37.197684 (+ 1993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:37.197687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:37.197704 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:37.197793 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:37.197799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.200050 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.200054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:37.201088 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:37.201096 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.206013 (+ 4917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.206016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:37.206111 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:37.206139 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 16:28:37.206193 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 16:28:37.206265 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1979,"thread_start_us":132,"threads_started":1}
I20260929 16:28:37.207396 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:37.194225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58788 (local address 127.0.98.3:36321)
0929 16:28:37.197242 (+ 3017us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:37.197250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:37.197273 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:28:37.197345 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:37.197360 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:37.197538 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:37.197941 (+ 403us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:37.198109 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.199187 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.201297 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.202111 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.206891 (+ 4780us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:37.207070 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:37.207136 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 16:28:37.207243 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2884,"thread_start_us":68,"threads_started":1}
I20260929 16:28:37.208448 13430 tablet_service.cc:1467] Processing CreateTablet for tablet 761d83fb2df04a6486997fa297ff3995 (DEFAULT_TABLE table=test-table [id=fc6be08a64f444ab8b811af6f4849f6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:37.209507 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:37.195190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42573 (local address 127.0.0.1:47418)
0929 16:28:37.196246 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:37.196257 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:37.196327 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:37.201188 (+ 4861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:37.201190 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:37.202213 (+ 1023us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:37.202303 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:37.202309 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.204046 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.204049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:37.207690 (+ 3641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:37.207703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.209065 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.209070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:37.209153 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:37.209178 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:28:37.209272 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 16:28:37.209364 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"thread_start_us":89,"threads_started":1}
I20260929 16:28:37.209726 13430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 761d83fb2df04a6486997fa297ff3995. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:37.212878 13565 tablet_service.cc:1467] Processing CreateTablet for tablet 761d83fb2df04a6486997fa297ff3995 (DEFAULT_TABLE table=test-table [id=fc6be08a64f444ab8b811af6f4849f6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:37.214135 13565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 761d83fb2df04a6486997fa297ff3995. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:37.225018 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:37.195799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47418 (local address 127.0.98.1:42573)
0929 16:28:37.199452 (+ 3653us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:37.199458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:37.199481 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:28:37.199550 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:37.199554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:37.199725 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:37.199871 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:37.202458 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.203535 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.207851 (+ 4316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.208693 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.208803 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:37.208933 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:37.224703 (+ 15770us) server_negotiation.cc:299] Negotiation successful
0929 16:28:37.224850 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3529,"thread_start_us":103,"threads_started":1}
I20260929 16:28:37.227093 13665 tablet_bootstrap.cc:492] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9: Bootstrap starting.
I20260929 16:28:37.228703 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 761d83fb2df04a6486997fa297ff3995 (DEFAULT_TABLE table=test-table [id=fc6be08a64f444ab8b811af6f4849f6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:37.229758 13665 tablet_bootstrap.cc:654] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:37.230176 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 761d83fb2df04a6486997fa297ff3995. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:37.230755 13665 log.cc:824] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:37.232978 13665 tablet_bootstrap.cc:492] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9: No bootstrap required, opened a new log
I20260929 16:28:37.233174 13665 ts_tablet_manager.cc:1397] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:28:37.234800 13667 tablet_bootstrap.cc:492] T 761d83fb2df04a6486997fa297ff3995 P 826654ec8c9a41fd97123de0216b44a5: Bootstrap starting.
I20260929 16:28:37.237519 13667 tablet_bootstrap.cc:654] T 761d83fb2df04a6486997fa297ff3995 P 826654ec8c9a41fd97123de0216b44a5: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:37.237516 13665 raft_consensus.cc:348] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676038e975bb4f3b88fd4b74fd06a7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36321 } } peers { permanent_uuid: "826654ec8c9a41fd97123de0216b44a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46441 } } peers { permanent_uuid: "a446ebbc1d634d8796f6f1414b79a72a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42573 } }
I20260929 16:28:37.238260 13665 raft_consensus.cc:374] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:37.238379 13665 raft_consensus.cc:729] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 676038e975bb4f3b88fd4b74fd06a7f9, State: Initialized, Role: FOLLOWER
I20260929 16:28:37.238415 13667 log.cc:824] T 761d83fb2df04a6486997fa297ff3995 P 826654ec8c9a41fd97123de0216b44a5: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:37.238977 13665 consensus_queue.cc:260] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676038e975bb4f3b88fd4b74fd06a7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36321 } } peers { permanent_uuid: "826654ec8c9a41fd97123de0216b44a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46441 } } peers { permanent_uuid: "a446ebbc1d634d8796f6f1414b79a72a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42573 } }
I20260929 16:28:37.239694 13631 heartbeater.cc:503] Master 127.0.98.62:36193 was elected leader, sending a full tablet report...
I20260929 16:28:37.243716 13665 ts_tablet_manager.cc:1428] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260929 16:28:37.245606 13669 tablet_bootstrap.cc:492] T 761d83fb2df04a6486997fa297ff3995 P a446ebbc1d634d8796f6f1414b79a72a: Bootstrap starting.
I20260929 16:28:37.248076 13669 tablet_bootstrap.cc:654] T 761d83fb2df04a6486997fa297ff3995 P a446ebbc1d634d8796f6f1414b79a72a: Neither blocks nor log segments found. Creating new log.
Sep 29 16:28:37 dist-test-slave-vq9v krb5kdc[12169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 16:28:37.249554 13669 log.cc:824] T 761d83fb2df04a6486997fa297ff3995 P a446ebbc1d634d8796f6f1414b79a72a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:37.252242 13667 tablet_bootstrap.cc:492] T 761d83fb2df04a6486997fa297ff3995 P 826654ec8c9a41fd97123de0216b44a5: No bootstrap required, opened a new log
I20260929 16:28:37.252241 13669 tablet_bootstrap.cc:492] T 761d83fb2df04a6486997fa297ff3995 P a446ebbc1d634d8796f6f1414b79a72a: No bootstrap required, opened a new log
I20260929 16:28:37.252499 13667 ts_tablet_manager.cc:1397] T 761d83fb2df04a6486997fa297ff3995 P 826654ec8c9a41fd97123de0216b44a5: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260929 16:28:37.252499 13669 ts_tablet_manager.cc:1397] T 761d83fb2df04a6486997fa297ff3995 P a446ebbc1d634d8796f6f1414b79a72a: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260929 16:28:37.257133 13667 raft_consensus.cc:348] T 761d83fb2df04a6486997fa297ff3995 P 826654ec8c9a41fd97123de0216b44a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676038e975bb4f3b88fd4b74fd06a7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36321 } } peers { permanent_uuid: "826654ec8c9a41fd97123de0216b44a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46441 } } peers { permanent_uuid: "a446ebbc1d634d8796f6f1414b79a72a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42573 } }
I20260929 16:28:37.257133 13669 raft_consensus.cc:348] T 761d83fb2df04a6486997fa297ff3995 P a446ebbc1d634d8796f6f1414b79a72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676038e975bb4f3b88fd4b74fd06a7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36321 } } peers { permanent_uuid: "826654ec8c9a41fd97123de0216b44a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46441 } } peers { permanent_uuid: "a446ebbc1d634d8796f6f1414b79a72a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42573 } }
I20260929 16:28:37.257470 13669 raft_consensus.cc:374] T 761d83fb2df04a6486997fa297ff3995 P a446ebbc1d634d8796f6f1414b79a72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:37.257663 13669 raft_consensus.cc:729] T 761d83fb2df04a6486997fa297ff3995 P a446ebbc1d634d8796f6f1414b79a72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a446ebbc1d634d8796f6f1414b79a72a, State: Initialized, Role: FOLLOWER
I20260929 16:28:37.258257 13669 consensus_queue.cc:260] T 761d83fb2df04a6486997fa297ff3995 P a446ebbc1d634d8796f6f1414b79a72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676038e975bb4f3b88fd4b74fd06a7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36321 } } peers { permanent_uuid: "826654ec8c9a41fd97123de0216b44a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46441 } } peers { permanent_uuid: "a446ebbc1d634d8796f6f1414b79a72a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42573 } }
I20260929 16:28:37.259465 13669 ts_tablet_manager.cc:1428] T 761d83fb2df04a6486997fa297ff3995 P a446ebbc1d634d8796f6f1414b79a72a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 16:28:37.259877 13363 heartbeater.cc:503] Master 127.0.98.62:36193 was elected leader, sending a full tablet report...
I20260929 16:28:37.260947 13667 raft_consensus.cc:374] T 761d83fb2df04a6486997fa297ff3995 P 826654ec8c9a41fd97123de0216b44a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:37.261036 13667 raft_consensus.cc:729] T 761d83fb2df04a6486997fa297ff3995 P 826654ec8c9a41fd97123de0216b44a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 826654ec8c9a41fd97123de0216b44a5, State: Initialized, Role: FOLLOWER
I20260929 16:28:37.261608 13667 consensus_queue.cc:260] T 761d83fb2df04a6486997fa297ff3995 P 826654ec8c9a41fd97123de0216b44a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676038e975bb4f3b88fd4b74fd06a7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36321 } } peers { permanent_uuid: "826654ec8c9a41fd97123de0216b44a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46441 } } peers { permanent_uuid: "a446ebbc1d634d8796f6f1414b79a72a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42573 } }
I20260929 16:28:37.265641 13497 heartbeater.cc:503] Master 127.0.98.62:36193 was elected leader, sending a full tablet report...
I20260929 16:28:37.266176 13667 ts_tablet_manager.cc:1428] T 761d83fb2df04a6486997fa297ff3995 P 826654ec8c9a41fd97123de0216b44a5: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
W20260929 16:28:37.272897 13364 tablet.cc:2369] T 761d83fb2df04a6486997fa297ff3995 P a446ebbc1d634d8796f6f1414b79a72a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:28:37.327445 13498 tablet.cc:2369] T 761d83fb2df04a6486997fa297ff3995 P 826654ec8c9a41fd97123de0216b44a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:28:37.371583 13632 tablet.cc:2369] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:28:37.408986 13670 raft_consensus.cc:482] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:28:37.409272 13670 raft_consensus.cc:504] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676038e975bb4f3b88fd4b74fd06a7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36321 } } peers { permanent_uuid: "826654ec8c9a41fd97123de0216b44a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46441 } } peers { permanent_uuid: "a446ebbc1d634d8796f6f1414b79a72a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42573 } }
I20260929 16:28:37.410741 13670 leader_election.cc:288] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 826654ec8c9a41fd97123de0216b44a5 (127.0.98.2:46441), a446ebbc1d634d8796f6f1414b79a72a (127.0.98.1:42573)
I20260929 16:28:37.422602 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:37.412329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51175 (local address 127.0.98.1:42573)
0929 16:28:37.412551 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:37.412555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:37.414260 (+ 1705us) server_negotiation.cc:407] Connection header received
0929 16:28:37.414442 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:37.414445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:37.414498 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:37.414547 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:37.415945 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.416736 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.419849 (+ 3113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.420556 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.420652 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:37.420729 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:37.422340 (+ 1611us) server_negotiation.cc:299] Negotiation successful
0929 16:28:37.422450 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260929 16:28:37.423331 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:37.411658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42573 (local address 127.0.98.3:51175)
0929 16:28:37.414178 (+ 2520us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:37.414193 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:37.414326 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:37.414678 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:37.414681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:37.414696 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:37.414815 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:37.414821 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.418487 (+ 3666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.418490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:37.419670 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:37.419684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.420836 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.420840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:37.420961 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:37.420985 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:28:37.423064 (+ 2079us) client_negotiation.cc:240] Negotiation successful
0929 16:28:37.423162 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2450,"thread_start_us":93,"threads_started":1}
I20260929 16:28:37.424028 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:37.412200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54765 (local address 127.0.98.2:46441)
0929 16:28:37.412460 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:37.412465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:37.412800 (+ 335us) server_negotiation.cc:407] Connection header received
0929 16:28:37.413029 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:37.413033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:37.413085 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:37.413132 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:37.414970 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.415779 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.421132 (+ 5353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.421802 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.421904 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:37.421970 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:37.423730 (+ 1760us) server_negotiation.cc:299] Negotiation successful
0929 16:28:37.423836 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260929 16:28:37.424628 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:37.411199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46441 (local address 127.0.98.3:54765)
0929 16:28:37.412707 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:37.412722 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:37.412886 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:37.413309 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:37.413312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:37.413329 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:37.413946 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:37.413954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.416907 (+ 2953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.416911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:37.418270 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:37.418313 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.422074 (+ 3761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.422077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:37.422184 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:37.422208 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:28:37.424386 (+ 2178us) client_negotiation.cc:240] Negotiation successful
0929 16:28:37.424482 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1440,"thread_start_us":132,"threads_started":1}
I20260929 16:28:37.425447 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761d83fb2df04a6486997fa297ff3995" candidate_uuid: "676038e975bb4f3b88fd4b74fd06a7f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a446ebbc1d634d8796f6f1414b79a72a" is_pre_election: true
I20260929 16:28:37.425824 13315 raft_consensus.cc:2454] T 761d83fb2df04a6486997fa297ff3995 P a446ebbc1d634d8796f6f1414b79a72a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 676038e975bb4f3b88fd4b74fd06a7f9 in term 0.
I20260929 16:28:37.426373 13435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761d83fb2df04a6486997fa297ff3995" candidate_uuid: "676038e975bb4f3b88fd4b74fd06a7f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826654ec8c9a41fd97123de0216b44a5" is_pre_election: true
I20260929 16:28:37.426409 13521 leader_election.cc:302] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 676038e975bb4f3b88fd4b74fd06a7f9, a446ebbc1d634d8796f6f1414b79a72a; no voters:
I20260929 16:28:37.426663 13435 raft_consensus.cc:2454] T 761d83fb2df04a6486997fa297ff3995 P 826654ec8c9a41fd97123de0216b44a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 676038e975bb4f3b88fd4b74fd06a7f9 in term 0.
I20260929 16:28:37.426738 13670 raft_consensus.cc:2781] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:28:37.426807 13670 raft_consensus.cc:482] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:28:37.426846 13670 raft_consensus.cc:3037] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:37.428102 13670 raft_consensus.cc:504] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676038e975bb4f3b88fd4b74fd06a7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36321 } } peers { permanent_uuid: "826654ec8c9a41fd97123de0216b44a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46441 } } peers { permanent_uuid: "a446ebbc1d634d8796f6f1414b79a72a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42573 } }
I20260929 16:28:37.428531 13670 leader_election.cc:288] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9 [CANDIDATE]: Term 1 election: Requested vote from peers 826654ec8c9a41fd97123de0216b44a5 (127.0.98.2:46441), a446ebbc1d634d8796f6f1414b79a72a (127.0.98.1:42573)
I20260929 16:28:37.429651 13435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761d83fb2df04a6486997fa297ff3995" candidate_uuid: "676038e975bb4f3b88fd4b74fd06a7f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "826654ec8c9a41fd97123de0216b44a5"
I20260929 16:28:37.429796 13435 raft_consensus.cc:3037] T 761d83fb2df04a6486997fa297ff3995 P 826654ec8c9a41fd97123de0216b44a5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:37.431221 13435 raft_consensus.cc:2454] T 761d83fb2df04a6486997fa297ff3995 P 826654ec8c9a41fd97123de0216b44a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 676038e975bb4f3b88fd4b74fd06a7f9 in term 1.
I20260929 16:28:37.431624 13520 leader_election.cc:302] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 676038e975bb4f3b88fd4b74fd06a7f9, 826654ec8c9a41fd97123de0216b44a5; no voters:
I20260929 16:28:37.431929 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761d83fb2df04a6486997fa297ff3995" candidate_uuid: "676038e975bb4f3b88fd4b74fd06a7f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a446ebbc1d634d8796f6f1414b79a72a"
I20260929 16:28:37.432089 13315 raft_consensus.cc:3037] T 761d83fb2df04a6486997fa297ff3995 P a446ebbc1d634d8796f6f1414b79a72a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:37.433154 13315 raft_consensus.cc:2454] T 761d83fb2df04a6486997fa297ff3995 P a446ebbc1d634d8796f6f1414b79a72a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 676038e975bb4f3b88fd4b74fd06a7f9 in term 1.
I20260929 16:28:37.433452 13670 raft_consensus.cc:2781] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:28:37.433699 13670 raft_consensus.cc:686] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9 [term 1 LEADER]: Becoming Leader. State: Replica: 676038e975bb4f3b88fd4b74fd06a7f9, State: Running, Role: LEADER
I20260929 16:28:37.434032 13670 consensus_queue.cc:237] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676038e975bb4f3b88fd4b74fd06a7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36321 } } peers { permanent_uuid: "826654ec8c9a41fd97123de0216b44a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46441 } } peers { permanent_uuid: "a446ebbc1d634d8796f6f1414b79a72a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42573 } }
I20260929 16:28:37.438024 13164 catalog_manager.cc:5495] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 676038e975bb4f3b88fd4b74fd06a7f9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "676038e975bb4f3b88fd4b74fd06a7f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676038e975bb4f3b88fd4b74fd06a7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "826654ec8c9a41fd97123de0216b44a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a446ebbc1d634d8796f6f1414b79a72a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42573 } health_report { overall_health: UNKNOWN } } }
I20260929 16:28:37.459777 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:37.454537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58802 (local address 127.0.98.3:36321)
0929 16:28:37.454659 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:37.454664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:37.454678 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:28:37.454767 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:37.454769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:37.454839 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:37.454955 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:28:37.455352 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.456138 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.457505 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:37.457813 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:37.457919 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:37.458353 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:28:37.458941 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:28:37.459072 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:37.459541 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 16:28:37.459608 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:28:37.467962 13315 raft_consensus.cc:1261] T 761d83fb2df04a6486997fa297ff3995 P a446ebbc1d634d8796f6f1414b79a72a [term 1 FOLLOWER]: Refusing update from remote peer 676038e975bb4f3b88fd4b74fd06a7f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:37.468967 13677 consensus_queue.cc:1035] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a446ebbc1d634d8796f6f1414b79a72a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:28:37.477896 13435 raft_consensus.cc:1261] T 761d83fb2df04a6486997fa297ff3995 P 826654ec8c9a41fd97123de0216b44a5 [term 1 FOLLOWER]: Refusing update from remote peer 676038e975bb4f3b88fd4b74fd06a7f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:37.478816 13677 consensus_queue.cc:1035] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "826654ec8c9a41fd97123de0216b44a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:28:37.498637 13680 mvcc.cc:205] Tried to move back new op lower bound from 7334704404338442240 to 7334704404213514240. Current Snapshot: MvccSnapshot[applied={T|T < 7334704404338442240}]
I20260929 16:28:37.508558 13683 mvcc.cc:205] Tried to move back new op lower bound from 7334704404338442240 to 7334704404213514240. Current Snapshot: MvccSnapshot[applied={T|T < 7334704404338442240}]
I20260929 16:28:37.508908 13681 mvcc.cc:205] Tried to move back new op lower bound from 7334704404338442240 to 7334704404213514240. Current Snapshot: MvccSnapshot[applied={T|T < 7334704404338442240}]
I20260929 16:28:37.514674 13164 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:28:37.514860 13164 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:28:37.517706 13164 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 502009154d5248d9b6a6b4d01cd550aa: Sending DeleteTablet for 3 replicas of tablet 761d83fb2df04a6486997fa297ff3995
I20260929 16:28:37.518723 13565 tablet_service.cc:1514] Processing DeleteTablet for tablet 761d83fb2df04a6486997fa297ff3995 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58788
I20260929 16:28:37.519554 13430 tablet_service.cc:1514] Processing DeleteTablet for tablet 761d83fb2df04a6486997fa297ff3995 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49052
I20260929 16:28:37.520560 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet 761d83fb2df04a6486997fa297ff3995 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47418
I20260929 16:28:37.521765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 13203
I20260929 16:28:37.524350 13690 tablet_replica.cc:330] T 761d83fb2df04a6486997fa297ff3995 P 826654ec8c9a41fd97123de0216b44a5: stopping tablet replica
I20260929 16:28:37.525614 13690 raft_consensus.cc:2229] T 761d83fb2df04a6486997fa297ff3995 P 826654ec8c9a41fd97123de0216b44a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:28:37.525960 13689 tablet_replica.cc:330] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9: stopping tablet replica
I20260929 16:28:37.526301 13689 raft_consensus.cc:2229] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:28:37.526731 13690 raft_consensus.cc:2258] T 761d83fb2df04a6486997fa297ff3995 P 826654ec8c9a41fd97123de0216b44a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:37.531762 13689 raft_consensus.cc:2258] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:37.534590 13689 ts_tablet_manager.cc:1905] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:28:37.535244 13690 ts_tablet_manager.cc:1905] T 761d83fb2df04a6486997fa297ff3995 P 826654ec8c9a41fd97123de0216b44a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:28:37.538254 13690 ts_tablet_manager.cc:1918] T 761d83fb2df04a6486997fa297ff3995 P 826654ec8c9a41fd97123de0216b44a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:28:37.540933 13689 ts_tablet_manager.cc:1918] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260929 16:28:37.541501 13152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42573 (error 108)
I20260929 16:28:37.542335 13689 log.cc:1196] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/761d83fb2df04a6486997fa297ff3995
I20260929 16:28:37.542559 13690 log.cc:1196] T 761d83fb2df04a6486997fa297ff3995 P 826654ec8c9a41fd97123de0216b44a5: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/761d83fb2df04a6486997fa297ff3995
I20260929 16:28:37.542801 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 13365
I20260929 16:28:37.545635 13689 ts_tablet_manager.cc:1939] T 761d83fb2df04a6486997fa297ff3995 P 676038e975bb4f3b88fd4b74fd06a7f9: Deleting consensus metadata
I20260929 16:28:37.550827 13149 catalog_manager.cc:4841] TS 676038e975bb4f3b88fd4b74fd06a7f9 (127.0.98.3:36321): tablet 761d83fb2df04a6486997fa297ff3995 (table test-table [id=fc6be08a64f444ab8b811af6f4849f6c]) successfully deleted
I20260929 16:28:37.552037 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:37.551688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42573 (local address 127.0.0.1:47424)
0929 16:28:37.551872 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:37.551948 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260929 16:28:37.552363 13152 catalog_manager.cc:4584] TS a446ebbc1d634d8796f6f1414b79a72a (127.0.98.1:42573): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 761d83fb2df04a6486997fa297ff3995: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42573: connect: Connection refused (error 111)
W20260929 16:28:37.555579 13663 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:28:37.554883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46441 (local address 127.0.0.1:49054)
0929 16:28:37.555047 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:37.555063 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:37.555153 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:37.555467 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46441: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 16:28:37.556255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 13500
I20260929 16:28:37.569455 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 13132
I20260929 16:28:37.578295 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 16:28:37.924172 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 16:28:37.924338 392 mini_ranger.cc:66] Stopping Ranger...
I20260929 16:28:38.269940 392 mini_ranger.cc:68] Stopped Ranger
2026-09-29 16:28:38.270 UTC [12199] LOG: received smart shutdown request
2026-09-29 16:28:38.274 UTC [12199] LOG: background worker "logical replication launcher" (PID 12208) exited with exit code 1
2026-09-29 16:28:38.274 UTC [12203] LOG: shutting down
2026-09-29 16:28:38.485 UTC [12199] LOG: database system is shut down
2026-09-29T16:28:38Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51803 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:28:38 dist-test-slave-vq9v krb5kdc[13702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:28:38 dist-test-slave-vq9v krb5kdc[13702](info): set up 2 sockets
Sep 29 16:28:38 dist-test-slave-vq9v krb5kdc[13702](info): commencing operation
krb5kdc: starting...
W20260929 16:28:39.681252 392 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:28:39 dist-test-slave-vq9v krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:28:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:28:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:28:39.816879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36503
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36503,127.0.98.61:36387,127.0.98.60:36493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:39.934125 13715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:39.934460 13715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:39.934538 13715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:39.939474 13715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:28:39.939586 13715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:39.939626 13715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:39.939666 13715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:28:39.939702 13715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:28:39.945510 13715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36503,127.0.98.61:36387,127.0.98.60:36493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:39.946733 13715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:39.947860 13715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:39.956915 13720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:39.956984 13723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:39.957134 13715 server_base.cc:1034] running on GCE node
W20260929 16:28:39.957013 13721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:39.958015 13715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:39.959203 13715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:39.960426 13715 hybrid_clock.cc:648] HybridClock initialized: now 1790699319960409 us; error 34 us; skew 500 ppm
Sep 29 16:28:39 dist-test-slave-vq9v krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:39.963945 13715 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:28:39.964982 13715 webserver.cc:460] Webserver started at http://127.0.98.62:43111/ using document root <none> and password file <none>
I20260929 16:28:39.965611 13715 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:39.965677 13715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:39.965909 13715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:39.968019 13715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "9148a3dcb23343fca9edc22446665622"
format_stamp: "Formatted at 2026-09-29 16:28:39 on dist-test-slave-vq9v"
server_key: "92dd4b24f3d120599e1729404b774b6d"
server_key_iv: "7d70a30f98b4085043d00aab3e3b12a5"
server_key_version: "encryptionkey@0"
I20260929 16:28:39.968608 13715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "9148a3dcb23343fca9edc22446665622"
format_stamp: "Formatted at 2026-09-29 16:28:39 on dist-test-slave-vq9v"
server_key: "92dd4b24f3d120599e1729404b774b6d"
server_key_iv: "7d70a30f98b4085043d00aab3e3b12a5"
server_key_version: "encryptionkey@0"
I20260929 16:28:39.972594 13715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 16:28:39.975566 13730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:39.977021 13715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:28:39.977260 13715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "9148a3dcb23343fca9edc22446665622"
format_stamp: "Formatted at 2026-09-29 16:28:39 on dist-test-slave-vq9v"
server_key: "92dd4b24f3d120599e1729404b774b6d"
server_key_iv: "7d70a30f98b4085043d00aab3e3b12a5"
server_key_version: "encryptionkey@0"
I20260929 16:28:39.977453 13715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:39.991719 13715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:39.995534 13715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:39.995750 13715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:40.003677 13715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36503
I20260929 16:28:40.003706 13782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36503 every 8 connection(s)
I20260929 16:28:40.004851 13715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:28:40.008169 13783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:40.012642 13783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:28:40.014541 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 13715
I20260929 16:28:40.014688 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:28:40.014986 392 external_mini_cluster.cc:1428] Setting key b8f7610ed9fb0a73b43d036a615d6147
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 29 16:28:40 dist-test-slave-vq9v krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:28:40.035418 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.015465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49250 (local address 127.0.98.62:36503)
0929 16:28:40.015967 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:40.015990 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:40.016346 (+ 356us) server_negotiation.cc:407] Connection header received
0929 16:28:40.018087 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:40.018092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:40.018322 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:40.018496 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:40.019672 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.021064 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.022281 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.022682 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.022882 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.028984 (+ 6102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:40.028991 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:40.029004 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:40.029034 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:40.031173 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.032002 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.032010 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.032013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.032166 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.032627 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.032630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.032631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.033066 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:40.033174 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:40.033821 (+ 647us) server_negotiation.cc:299] Negotiation successful
0929 16:28:40.034269 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":319,"spinlock_wait_cycles":5888,"thread_start_us":157,"threads_started":1}
I20260929 16:28:40.035418 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.015466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36503 (local address 127.0.0.1:49250)
0929 16:28:40.016171 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.016214 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:40.017674 (+ 1460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:40.018574 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:40.018579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:40.019163 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:40.019482 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:40.019499 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.021337 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.021341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:40.022136 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:40.022146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.022866 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.022870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:40.022982 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.023689 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:40.023709 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:40.028739 (+ 5030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:40.031330 (+ 2591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:40.031345 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:40.031360 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:40.031749 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:40.032297 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:40.032303 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:40.032306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:40.032515 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:40.033231 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:40.033241 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:40.033471 (+ 230us) client_negotiation.cc:769] Sending connection context
0929 16:28:40.033729 (+ 258us) client_negotiation.cc:240] Negotiation successful
0929 16:28:40.034304 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":206,"spinlock_wait_cycles":896,"thread_start_us":199,"threads_started":1}
I20260929 16:28:40.038273 13783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:28:40.039340 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.038941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36387 (local address 127.0.0.1:36416)
0929 16:28:40.039159 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.039251 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260929 16:28:40.039587 13732 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36387: connect: Connection refused (error 111)
I20260929 16:28:40.040702 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.040434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36387 (local address 127.0.0.1:36418)
0929 16:28:40.040599 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.040646 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 16:28:40.040915 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36387: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36387: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 16:28:40.044504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36387
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36503,127.0.98.61:36387,127.0.98.60:36493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:28:40.090173 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } attempt: 1
I20260929 16:28:40.091238 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.090899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36387 (local address 127.0.0.1:36424)
0929 16:28:40.091080 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.091152 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 16:28:40.092211 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.091926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36387 (local address 127.0.0.1:36430)
0929 16:28:40.092067 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.092134 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 16:28:40.092423 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36387: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36387: connect: Connection refused (error 111)
I20260929 16:28:40.160596 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } attempt: 2
I20260929 16:28:40.161572 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.161148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36387 (local address 127.0.0.1:36432)
0929 16:28:40.161415 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.161487 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260929 16:28:40.162629 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.162394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36387 (local address 127.0.0.1:36444)
0929 16:28:40.162504 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.162556 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 16:28:40.162909 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36387: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36387: connect: Connection refused (error 111)
W20260929 16:28:40.165767 13790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:40.166050 13790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:40.166126 13790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:40.172112 13790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:28:40.172221 13790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:40.172259 13790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:40.172297 13790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:28:40.172320 13790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:28:40.178571 13790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36503,127.0.98.61:36387,127.0.98.60:36493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:40.179739 13790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:40.180861 13790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:40.190035 13796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:40.190042 13798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:40.190042 13795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:40.190109 13790 server_base.cc:1034] running on GCE node
I20260929 16:28:40.191073 13790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:40.192317 13790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:40.193535 13790 hybrid_clock.cc:648] HybridClock initialized: now 1790699320193513 us; error 34 us; skew 500 ppm
Sep 29 16:28:40 dist-test-slave-vq9v krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:40.196605 13790 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 16:28:40.197624 13790 webserver.cc:460] Webserver started at http://127.0.98.61:40817/ using document root <none> and password file <none>
I20260929 16:28:40.198205 13790 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:40.198303 13790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:40.198540 13790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:40.200663 13790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-1/data/instance:
uuid: "0918733557a1417fb6f604a5bd8bbf0e"
format_stamp: "Formatted at 2026-09-29 16:28:40 on dist-test-slave-vq9v"
server_key: "3cb4d747d9afc9345653e778f66eeda3"
server_key_iv: "715bd07c3084f4f6a39a90596d1ae3cf"
server_key_version: "encryptionkey@0"
I20260929 16:28:40.201362 13790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-1/wal/instance:
uuid: "0918733557a1417fb6f604a5bd8bbf0e"
format_stamp: "Formatted at 2026-09-29 16:28:40 on dist-test-slave-vq9v"
server_key: "3cb4d747d9afc9345653e778f66eeda3"
server_key_iv: "715bd07c3084f4f6a39a90596d1ae3cf"
server_key_version: "encryptionkey@0"
I20260929 16:28:40.206485 13790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 16:28:40.210657 13805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:40.212730 13790 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:28:40.212929 13790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-1/wal
uuid: "0918733557a1417fb6f604a5bd8bbf0e"
format_stamp: "Formatted at 2026-09-29 16:28:40 on dist-test-slave-vq9v"
server_key: "3cb4d747d9afc9345653e778f66eeda3"
server_key_iv: "715bd07c3084f4f6a39a90596d1ae3cf"
server_key_version: "encryptionkey@0"
I20260929 16:28:40.213107 13790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:40.251571 13790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:40.254129 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } attempt: 3
I20260929 16:28:40.255044 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.254680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36387 (local address 127.0.0.1:36450)
0929 16:28:40.254861 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.254939 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 16:28:40.255764 13790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:40.255889 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.255613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36387 (local address 127.0.0.1:36458)
0929 16:28:40.255744 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.255805 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 16:28:40.255990 13790 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260929 16:28:40.256074 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36387: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36387: connect: Connection refused (error 111)
I20260929 16:28:40.264074 13790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36387
I20260929 16:28:40.264094 13857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36387 every 8 connection(s)
I20260929 16:28:40.265259 13790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-1/data/info.pb
I20260929 16:28:40.268126 13858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:40.272089 13858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:28:40.273420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 13790
I20260929 16:28:40.273586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-1/wal/instance
I20260929 16:28:40.274005 392 external_mini_cluster.cc:1428] Setting key 169efd6df385e31e7c79cd52dc44c789
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 29 16:28:40 dist-test-slave-vq9v krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:28:40.291990 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.274424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49256 (local address 127.0.98.62:36503)
0929 16:28:40.274571 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:40.274575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:40.275740 (+ 1165us) server_negotiation.cc:407] Connection header received
0929 16:28:40.277382 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:40.277387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:40.277458 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:40.277551 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:40.279399 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.280418 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.281943 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.282388 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.282525 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.287985 (+ 5460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:40.287989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:40.287991 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:40.288021 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:40.289672 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.290280 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.290284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.290285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.290369 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.290751 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.290755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.290757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.290963 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:40.291070 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:40.291650 (+ 580us) server_negotiation.cc:299] Negotiation successful
0929 16:28:40.291791 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:28:40.292829 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.274954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36503 (local address 127.0.0.1:49256)
0929 16:28:40.275547 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.275595 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:40.277067 (+ 1472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:40.277795 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:40.277808 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:40.278377 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:40.279175 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:40.279199 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.280605 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.280608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:40.281762 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:40.281772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.282630 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.282636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:40.282894 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.283623 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:40.283658 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:40.287728 (+ 4070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:40.289828 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:40.289835 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:40.289853 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:40.290152 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:40.290465 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:40.290469 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:40.290471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:40.290644 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:40.291070 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:40.291078 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:40.291371 (+ 293us) client_negotiation.cc:769] Sending connection context
0929 16:28:40.291578 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 16:28:40.291950 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":182,"threads_started":1}
I20260929 16:28:40.294517 13858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:28:40 dist-test-slave-vq9v krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 16:28:40.305840 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36493
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36503,127.0.98.61:36387,127.0.98.60:36493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:28:40.306468 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.295103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36387 (local address 127.0.0.1:36468)
0929 16:28:40.295266 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.295281 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:40.295374 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:40.296258 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:40.296261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:40.296473 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:40.296582 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:40.296588 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.297849 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.297852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:40.298525 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:40.298534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.299367 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.299371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:40.299459 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.300088 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:40.300100 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:40.301670 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:40.304322 (+ 2652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:40.304325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:40.304327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:40.304658 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:40.305039 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:40.305046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:40.305051 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:40.305150 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:40.305873 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:40.305875 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:40.305970 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 16:28:40.306129 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 16:28:40.306319 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 16:28:40.306754 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.295272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36468 (local address 127.0.98.61:36387)
0929 16:28:40.295642 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:40.295656 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:40.295683 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:28:40.295737 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:40.295742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:40.295974 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:40.296092 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:40.296710 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.297724 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.298661 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.299102 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.299200 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.301880 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:40.301891 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:40.301905 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:40.301929 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:40.304150 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.304829 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.304835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.304836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.304925 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.305326 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.305330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.305335 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.305732 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:40.305890 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:40.306264 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 16:28:40.306448 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":144,"threads_started":1}
I20260929 16:28:40.309002 13858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:28:40.309966 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.309598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36493 (local address 127.0.0.1:41386)
0929 16:28:40.309745 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.309843 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 16:28:40.310184 13808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36493: connect: Connection refused (error 111)
I20260929 16:28:40.311131 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.310874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36493 (local address 127.0.0.1:41402)
0929 16:28:40.311018 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.311072 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 16:28:40.311313 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36493: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36493: connect: Connection refused (error 111)
I20260929 16:28:40.360597 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } attempt: 1
I20260929 16:28:40.361650 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.361273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36493 (local address 127.0.0.1:41404)
0929 16:28:40.361491 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.361567 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260929 16:28:40.362411 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.362168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36493 (local address 127.0.0.1:41412)
0929 16:28:40.362291 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.362349 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 16:28:40.362551 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36493: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36493: connect: Connection refused (error 111)
I20260929 16:28:40.399317 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } attempt: 4
Sep 29 16:28:40 dist-test-slave-vq9v krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 16:28:40.411046 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.400119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36478 (local address 127.0.98.61:36387)
0929 16:28:40.400289 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:40.400296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:40.400327 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:28:40.400453 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:40.400456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:40.400520 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:40.400633 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:40.401547 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.402969 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.404195 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.404613 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.404756 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.407474 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:40.407477 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:40.407480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:40.407508 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:40.409165 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.409756 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.409761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.409762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.409851 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.410151 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.410156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.410157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.410365 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:40.410478 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:40.410754 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 16:28:40.410876 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 16:28:40.411046 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.399971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36387 (local address 127.0.0.1:36478)
0929 16:28:40.400221 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.400236 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:40.400339 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:40.400664 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:40.400665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:40.400858 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:40.401383 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:40.401392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.403317 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.403320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:40.404010 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:40.404023 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.404843 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.404846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:40.404951 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.405669 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:40.405684 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:40.407254 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:40.409398 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:40.409403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:40.409408 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:40.409645 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:40.409960 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:40.409964 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:40.409966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:40.410055 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:40.410467 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:40.410470 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:40.410582 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 16:28:40.410690 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 16:28:40.410869 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260929 16:28:40.411929 13783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:28:40.412765 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.412491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36493 (local address 127.0.0.1:41414)
0929 16:28:40.412636 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.412693 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 16:28:40.413546 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.413326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36493 (local address 127.0.0.1:41424)
0929 16:28:40.413434 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.413491 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 16:28:40.413674 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36493: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36493: connect: Connection refused (error 111)
W20260929 16:28:40.427456 13864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:40.427846 13864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:40.427953 13864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 16:28:40.430751 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } attempt: 2
I20260929 16:28:40.431689 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.431369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36493 (local address 127.0.0.1:41432)
0929 16:28:40.431527 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.431604 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 16:28:40.432505 13864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20260929 16:28:40.432521 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.432265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36493 (local address 127.0.0.1:41444)
0929 16:28:40.432395 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.432461 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 16:28:40.432626 13864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:40.432677 13864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:40.432677 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36493: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36493: connect: Connection refused (error 111)
W20260929 16:28:40.432735 13864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:28:40.432783 13864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:28:40.438870 13864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36503,127.0.98.61:36387,127.0.98.60:36493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:40.440104 13864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:40.441310 13864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:40.450471 13873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:40.450471 13870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:40.450760 13864 server_base.cc:1034] running on GCE node
W20260929 16:28:40.450470 13871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:40.451483 13864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:40.452720 13864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:40.453922 13864 hybrid_clock.cc:648] HybridClock initialized: now 1790699320453898 us; error 30 us; skew 500 ppm
Sep 29 16:28:40 dist-test-slave-vq9v krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:40.457378 13864 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 16:28:40.458423 13864 webserver.cc:460] Webserver started at http://127.0.98.60:40369/ using document root <none> and password file <none>
I20260929 16:28:40.459043 13864 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:40.459139 13864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:40.459383 13864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:40.461562 13864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-2/data/instance:
uuid: "a2c791357c0f445da36dfef65a3155ca"
format_stamp: "Formatted at 2026-09-29 16:28:40 on dist-test-slave-vq9v"
server_key: "a74de6f7fd062d7ba83a8e761a01c270"
server_key_iv: "b73166771cf77bc928333f1133290606"
server_key_version: "encryptionkey@0"
I20260929 16:28:40.462149 13864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-2/wal/instance:
uuid: "a2c791357c0f445da36dfef65a3155ca"
format_stamp: "Formatted at 2026-09-29 16:28:40 on dist-test-slave-vq9v"
server_key: "a74de6f7fd062d7ba83a8e761a01c270"
server_key_iv: "b73166771cf77bc928333f1133290606"
server_key_version: "encryptionkey@0"
I20260929 16:28:40.466037 13864 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:28:40.468833 13880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:40.470187 13864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:28:40.470341 13864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-2/wal
uuid: "a2c791357c0f445da36dfef65a3155ca"
format_stamp: "Formatted at 2026-09-29 16:28:40 on dist-test-slave-vq9v"
server_key: "a74de6f7fd062d7ba83a8e761a01c270"
server_key_iv: "b73166771cf77bc928333f1133290606"
server_key_version: "encryptionkey@0"
I20260929 16:28:40.470479 13864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:40.472849 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } attempt: 1
I20260929 16:28:40.473822 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.473482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36493 (local address 127.0.0.1:41458)
0929 16:28:40.473660 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.473729 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 16:28:40.474727 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.474458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36493 (local address 127.0.0.1:41460)
0929 16:28:40.474587 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.474644 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 16:28:40.474888 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36493: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36493: connect: Connection refused (error 111)
I20260929 16:28:40.491039 13864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:40.494652 13864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:40.494813 13864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:40.502815 13864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36493
I20260929 16:28:40.503067 13932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36493 every 8 connection(s)
I20260929 16:28:40.503988 13864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-2/data/info.pb
I20260929 16:28:40.507270 13933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:40.511314 13933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:28:40.513823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 13864
I20260929 16:28:40.513962 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790699152396020-392-0/minicluster-data/master-2/wal/instance
I20260929 16:28:40.514282 392 external_mini_cluster.cc:1428] Setting key 8d67ccddd72c07518210a45c302be85a
Sep 29 16:28:40 dist-test-slave-vq9v krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 16:28:40 dist-test-slave-vq9v krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 16:28:40.523878 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } attempt: 3
I20260929 16:28:40.526918 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.513394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49258 (local address 127.0.98.62:36503)
0929 16:28:40.513551 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:40.513555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:40.514571 (+ 1016us) server_negotiation.cc:407] Connection header received
0929 16:28:40.515784 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:40.515788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:40.515836 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:40.515942 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:40.517128 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.517707 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.519065 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.519370 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.519484 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.522470 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:40.522474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:40.522476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:40.522502 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:40.524353 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.525017 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.525021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.525023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.525140 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.525600 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.525606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.525608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.525899 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:40.526017 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:40.526606 (+ 589us) server_negotiation.cc:299] Negotiation successful
0929 16:28:40.526735 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:28:40.528016 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.513801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36503 (local address 127.0.0.1:49258)
0929 16:28:40.514380 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.514446 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:40.515585 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:40.516051 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:40.516057 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:40.516545 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:40.516984 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:40.516997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.517817 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.517824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:40.518923 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:40.518933 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.519523 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.519526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:40.519672 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.520310 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:40.520336 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:40.522283 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:40.524506 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:40.524513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:40.524524 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:40.524902 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:40.525326 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:40.525331 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:40.525332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:40.525482 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:40.526023 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:40.526030 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:40.526353 (+ 323us) client_negotiation.cc:769] Sending connection context
0929 16:28:40.526559 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 16:28:40.526889 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":156,"threads_started":1}
I20260929 16:28:40.528031 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.515020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41474 (local address 127.0.98.60:36493)
0929 16:28:40.515415 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:40.515424 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:40.515458 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:28:40.515582 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:40.515596 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:40.515837 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:40.516018 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:40.516965 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.518196 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.519262 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.519796 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.519879 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.522967 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:40.522972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:40.522983 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:40.523009 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:40.525107 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.525696 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.525704 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.525709 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.525804 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.526124 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.526127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.526130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.526439 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:40.527452 (+ 1013us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:40.527571 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 16:28:40.527755 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":143,"threads_started":1}
Sep 29 16:28:40 dist-test-slave-vq9v krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 16:28:40.530246 13933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:28:40.535060 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.524443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36493 (local address 127.0.0.1:41480)
0929 16:28:40.524668 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.524687 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:40.524789 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:40.525239 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:40.525245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:40.525484 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:40.525620 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:40.525632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.526707 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.526711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:40.527523 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:40.527537 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.528174 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.528180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:40.528280 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.528872 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:40.528884 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:40.530320 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:40.533370 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:40.533373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:40.533375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:40.533670 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:40.533994 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:40.533997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:40.533999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:40.534091 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:40.534551 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:40.534554 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:40.534643 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 16:28:40.534721 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 16:28:40.534856 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260929 16:28:40.536010 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.524482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41480 (local address 127.0.98.60:36493)
0929 16:28:40.524922 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:40.524931 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:40.524943 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:28:40.525002 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:40.525006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:40.525054 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:40.525159 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:40.525756 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.526547 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.527682 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.528021 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.528145 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.530517 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:40.530521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:40.530523 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:40.530551 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:40.533165 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.533782 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.533786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.533788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.533882 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.534184 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.534189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.534191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.534441 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:40.535553 (+ 1112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:40.535658 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 16:28:40.535782 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":124,"threads_started":1}
Sep 29 16:28:40 dist-test-slave-vq9v krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 16:28:40 dist-test-slave-vq9v krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:28:40.539960 13858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e: Bootstrap starting.
I20260929 16:28:40.542040 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } attempt: 2
I20260929 16:28:40.542243 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.531391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49266 (local address 127.0.98.62:36503)
0929 16:28:40.531532 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:40.531536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:40.531546 (+ 10us) server_negotiation.cc:407] Connection header received
0929 16:28:40.531581 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:40.531583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:40.531660 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:40.531753 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:40.532595 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.533422 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.534574 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.534816 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.534941 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.538373 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:40.538378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:40.538381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:40.538420 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:40.540101 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.540649 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.540653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.540655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.540750 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.541085 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.541092 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.541096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.541409 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:40.541660 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:40.541826 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 16:28:40.541965 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:28:40.542531 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.530922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36494 (local address 127.0.98.61:36387)
0929 16:28:40.531087 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:40.531093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:40.531464 (+ 371us) server_negotiation.cc:407] Connection header received
0929 16:28:40.531668 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:40.531672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:40.531726 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:40.531805 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:40.532372 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.533188 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.534441 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.534765 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.534854 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.537589 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:40.537593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:40.537596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:40.537624 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:40.539435 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.540683 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.540687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.540689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.540794 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.541121 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.541124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.541126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.541470 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:40.541568 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:40.541916 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 16:28:40.542245 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:28:40.543051 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.530757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36387 (local address 127.0.0.1:36494)
0929 16:28:40.531384 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.531399 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:40.531542 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:40.531935 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:40.531937 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:40.532129 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:40.532238 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:40.532244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.533373 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.533379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:40.534279 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:40.534291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.534994 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.534997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:40.535112 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.535783 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:40.535797 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:40.537438 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:40.540274 (+ 2836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:40.540279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:40.540284 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:40.540543 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:40.540903 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:40.540908 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:40.540910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:40.541009 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:40.541615 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:40.541619 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:40.541723 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 16:28:40.542712 (+ 989us) client_negotiation.cc:240] Negotiation successful
0929 16:28:40.542874 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541}
I20260929 16:28:40.543866 13933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:28:40.544354 13858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:40.545460 13858 log.cc:824] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:40.548611 13858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e: No bootstrap required, opened a new log
Sep 29 16:28:40 dist-test-slave-vq9v krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 16:28:40 dist-test-slave-vq9v krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 16:28:40 dist-test-slave-vq9v krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 16:28:40.554069 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.544306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36493 (local address 127.0.0.1:41502)
0929 16:28:40.544502 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.544519 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:40.544599 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:40.544923 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:40.544927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:40.545113 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:40.545281 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:40.545293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.546253 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.546256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:40.547057 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:40.547065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.547636 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.547639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:40.547712 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.548262 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:40.548275 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:40.549853 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:40.551932 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:40.551938 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:40.551943 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:40.552197 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:40.552513 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:40.552519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:40.552523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:40.552612 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:40.553249 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:40.553252 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:40.553357 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 16:28:40.553454 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 16:28:40.553867 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 16:28:40.554299 13858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9148a3dcb23343fca9edc22446665622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "0918733557a1417fb6f604a5bd8bbf0e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } } peers { permanent_uuid: "a2c791357c0f445da36dfef65a3155ca" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } }
I20260929 16:28:40.554591 13858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:40.554664 13858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0918733557a1417fb6f604a5bd8bbf0e, State: Initialized, Role: FOLLOWER
I20260929 16:28:40.555002 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.544563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41502 (local address 127.0.98.60:36493)
0929 16:28:40.544681 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:40.544687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:40.544702 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:28:40.544757 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:40.544759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:40.544799 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:40.544880 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:40.545439 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.546156 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.547171 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.547536 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.547937 (+ 401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.549991 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:40.549995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:40.549998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:40.550019 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:40.551827 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.552316 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.552320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.552322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.552411 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.552732 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.552737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.552740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.553105 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:40.554540 (+ 1435us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:40.554689 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 16:28:40.554846 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:28:40.555281 13858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9148a3dcb23343fca9edc22446665622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "0918733557a1417fb6f604a5bd8bbf0e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } } peers { permanent_uuid: "a2c791357c0f445da36dfef65a3155ca" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } }
I20260929 16:28:40.555971 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.543957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.0.98.61:36387)
0929 16:28:40.544067 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:40.544072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:40.544121 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:28:40.544293 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:40.544297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:40.544383 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:40.544704 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:40.545694 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.546397 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.547458 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.547760 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.547850 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.551698 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:40.551703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:40.551706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:40.551741 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:40.553710 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.554317 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.554324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.554326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.554422 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.554791 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.554794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.554796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.555031 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:40.555202 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:40.555685 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 16:28:40.555820 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:28:40.556376 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.542504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36493 (local address 127.0.0.1:41486)
0929 16:28:40.542878 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:40.542893 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:40.543052 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:40.546146 (+ 3094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:40.546150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:40.546343 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:40.546451 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:40.546459 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.547523 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.547526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:40.548379 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:40.548391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.549058 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.549062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:40.549165 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.549685 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:40.549701 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:40.551403 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:40.553981 (+ 2578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:40.553985 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:40.553987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:40.554270 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:40.554845 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:40.554848 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:40.554850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:40.554953 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:40.555806 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:40.555808 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:40.555927 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 16:28:40.556013 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 16:28:40.556185 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306}
I20260929 16:28:40.556952 13858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:28:40.557173 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.542613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41486 (local address 127.0.98.60:36493)
0929 16:28:40.542752 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:40.542757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:40.545932 (+ 3175us) server_negotiation.cc:407] Connection header received
0929 16:28:40.545972 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:40.545974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:40.546011 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:40.546569 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:40.546655 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.547394 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.548532 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.548898 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.549426 (+ 528us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.551566 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:40.551573 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:40.551577 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:40.551606 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:40.553851 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.554536 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.554539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.554540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.554695 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.555187 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.555193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.555195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.555694 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:40.556806 (+ 1112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:40.556879 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 16:28:40.557000 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:28:40.558382 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9148a3dcb23343fca9edc22446665622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "0918733557a1417fb6f604a5bd8bbf0e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } } peers { permanent_uuid: "a2c791357c0f445da36dfef65a3155ca" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } } }
I20260929 16:28:40.558584 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:28:40.559055 13933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca: Bootstrap starting.
I20260929 16:28:40.560554 13783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622: Bootstrap starting.
I20260929 16:28:40.560637 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:28:40.562237 13933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:40.563164 13933 log.cc:824] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:40.563356 13783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:40.564126 13783 log.cc:824] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:40.565646 13933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca: No bootstrap required, opened a new log
I20260929 16:28:40.566959 13783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622: No bootstrap required, opened a new log
W20260929 16:28:40.564409 13951 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:28:40.567159 13951 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:28:40.570082 13933 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9148a3dcb23343fca9edc22446665622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "0918733557a1417fb6f604a5bd8bbf0e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } } peers { permanent_uuid: "a2c791357c0f445da36dfef65a3155ca" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } }
I20260929 16:28:40.570362 13933 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:40.570441 13933 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c791357c0f445da36dfef65a3155ca, State: Initialized, Role: FOLLOWER
I20260929 16:28:40.566516 13790 master_runner.cc:186] Error getting master registration for 127.0.98.62:36503: OK, instance_id { permanent_uuid: "9148a3dcb23343fca9edc22446665622" instance_seqno: 1790699320001171 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:28:40.570713 13790 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:28:40.570997 13933 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9148a3dcb23343fca9edc22446665622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "0918733557a1417fb6f604a5bd8bbf0e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } } peers { permanent_uuid: "a2c791357c0f445da36dfef65a3155ca" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } }
I20260929 16:28:40.571897 13783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9148a3dcb23343fca9edc22446665622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "0918733557a1417fb6f604a5bd8bbf0e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } } peers { permanent_uuid: "a2c791357c0f445da36dfef65a3155ca" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } }
I20260929 16:28:40.572186 13783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:40.572250 13783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9148a3dcb23343fca9edc22446665622, State: Initialized, Role: FOLLOWER
I20260929 16:28:40.572444 13933 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:28:40.572815 13783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9148a3dcb23343fca9edc22446665622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "0918733557a1417fb6f604a5bd8bbf0e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } } peers { permanent_uuid: "a2c791357c0f445da36dfef65a3155ca" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } }
I20260929 16:28:40.574892 13783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:28:40.575196 13963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9148a3dcb23343fca9edc22446665622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "0918733557a1417fb6f604a5bd8bbf0e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } } peers { permanent_uuid: "a2c791357c0f445da36dfef65a3155ca" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } } }
I20260929 16:28:40.575477 13963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:28:40.577348 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.566091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49278 (local address 127.0.98.62:36503)
0929 16:28:40.566208 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:40.566213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:40.567023 (+ 810us) server_negotiation.cc:407] Connection header received
0929 16:28:40.567345 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:40.567348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:40.567406 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:40.567479 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:40.568561 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.569371 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.570664 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.570985 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.571062 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.573182 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:40.573187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:40.573189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:40.573245 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:40.575049 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.575813 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.575817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.575831 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.575921 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.576238 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.576242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.576244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.576473 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:40.576598 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:40.576989 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 16:28:40.577111 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 16:28:40.579708 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9148a3dcb23343fca9edc22446665622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "0918733557a1417fb6f604a5bd8bbf0e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } } peers { permanent_uuid: "a2c791357c0f445da36dfef65a3155ca" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } } }
I20260929 16:28:40.580384 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca [sys.catalog]: This master's current role is: FOLLOWER
W20260929 16:28:40.585668 13985 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:28:40.585757 13985 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:28:40.587160 13864 master_runner.cc:263] No leader master found from master a2c791357c0f445da36dfef65a3155ca
I20260929 16:28:40.587229 13864 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:28:40.588165 13715 master_runner.cc:263] No leader master found from master 9148a3dcb23343fca9edc22446665622
I20260929 16:28:40.588261 13715 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:28:40.593580 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.566010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41504 (local address 127.0.98.60:36493)
0929 16:28:40.570585 (+ 4575us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:40.570590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:40.575260 (+ 4670us) server_negotiation.cc:407] Connection header received
0929 16:28:40.575512 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:40.575517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:40.575595 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:40.575646 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:40.578521 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.579285 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.580836 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.581145 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.581262 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.585950 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:40.585954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:40.585957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:40.585985 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:40.587975 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.590875 (+ 2900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.590880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.590882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.590991 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.591742 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.591746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.591748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.592005 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:40.592143 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:40.593246 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0929 16:28:40.593387 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4465}
I20260929 16:28:40.594772 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:40.566513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36514 (local address 127.0.98.61:36387)
0929 16:28:40.570858 (+ 4345us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:40.570863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:40.578374 (+ 7511us) server_negotiation.cc:407] Connection header received
0929 16:28:40.578490 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:40.578494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:40.578548 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:40.578629 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:40.579374 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.580155 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.581386 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:40.581718 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:40.581797 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:40.588528 (+ 6731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:40.588533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:40.588535 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:40.588564 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:40.590329 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.591328 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.591332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.591334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.591423 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:40.592492 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:40.592496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:40.592497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:40.592719 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:40.592845 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:40.594480 (+ 1635us) server_negotiation.cc:299] Negotiation successful
0929 16:28:40.594610 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4266}
W20260929 16:28:40.597098 13983 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:28:40.597344 13983 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:28:40.674170 13963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:28:40.674415 13963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9148a3dcb23343fca9edc22446665622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "0918733557a1417fb6f604a5bd8bbf0e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } } peers { permanent_uuid: "a2c791357c0f445da36dfef65a3155ca" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } }
I20260929 16:28:40.675866 13963 leader_election.cc:288] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0918733557a1417fb6f604a5bd8bbf0e (127.0.98.61:36387), a2c791357c0f445da36dfef65a3155ca (127.0.98.60:36493)
I20260929 16:28:40.676776 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9148a3dcb23343fca9edc22446665622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0918733557a1417fb6f604a5bd8bbf0e" is_pre_election: true
I20260929 16:28:40.677811 13830 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9148a3dcb23343fca9edc22446665622 in term 0.
I20260929 16:28:40.678745 13907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9148a3dcb23343fca9edc22446665622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c791357c0f445da36dfef65a3155ca" is_pre_election: true
I20260929 16:28:40.679080 13907 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9148a3dcb23343fca9edc22446665622 in term 0.
I20260929 16:28:40.679706 13732 leader_election.cc:302] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0918733557a1417fb6f604a5bd8bbf0e, 9148a3dcb23343fca9edc22446665622; no voters:
I20260929 16:28:40.680064 13963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:28:40.680161 13963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:28:40.680213 13963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:40.681684 13963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9148a3dcb23343fca9edc22446665622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "0918733557a1417fb6f604a5bd8bbf0e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } } peers { permanent_uuid: "a2c791357c0f445da36dfef65a3155ca" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } }
I20260929 16:28:40.682090 13963 leader_election.cc:288] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [CANDIDATE]: Term 1 election: Requested vote from peers 0918733557a1417fb6f604a5bd8bbf0e (127.0.98.61:36387), a2c791357c0f445da36dfef65a3155ca (127.0.98.60:36493)
I20260929 16:28:40.682934 13907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9148a3dcb23343fca9edc22446665622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c791357c0f445da36dfef65a3155ca"
I20260929 16:28:40.683086 13907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:40.684477 13907 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9148a3dcb23343fca9edc22446665622 in term 1.
I20260929 16:28:40.685031 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9148a3dcb23343fca9edc22446665622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0918733557a1417fb6f604a5bd8bbf0e"
I20260929 16:28:40.685182 13830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:40.686635 13830 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9148a3dcb23343fca9edc22446665622 in term 1.
I20260929 16:28:40.687064 13732 leader_election.cc:302] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0918733557a1417fb6f604a5bd8bbf0e, 9148a3dcb23343fca9edc22446665622; no voters:
I20260929 16:28:40.687443 13963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:28:40.687709 13963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [term 1 LEADER]: Becoming Leader. State: Replica: 9148a3dcb23343fca9edc22446665622, State: Running, Role: LEADER
I20260929 16:28:40.688102 13963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9148a3dcb23343fca9edc22446665622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "0918733557a1417fb6f604a5bd8bbf0e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } } peers { permanent_uuid: "a2c791357c0f445da36dfef65a3155ca" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } }
I20260929 16:28:40.691679 13963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9148a3dcb23343fca9edc22446665622. Latest consensus state: current_term: 1 leader_uuid: "9148a3dcb23343fca9edc22446665622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9148a3dcb23343fca9edc22446665622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "0918733557a1417fb6f604a5bd8bbf0e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } } peers { permanent_uuid: "a2c791357c0f445da36dfef65a3155ca" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } } }
I20260929 16:28:40.691828 13963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:40.693817 13990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:28:40.694736 13990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:28:40.702577 13907 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca [term 1 FOLLOWER]: Refusing update from remote peer 9148a3dcb23343fca9edc22446665622: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:40.705921 13963 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c791357c0f445da36dfef65a3155ca" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:28:40.711571 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9148a3dcb23343fca9edc22446665622. Latest consensus state: current_term: 1 leader_uuid: "9148a3dcb23343fca9edc22446665622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9148a3dcb23343fca9edc22446665622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "0918733557a1417fb6f604a5bd8bbf0e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } } peers { permanent_uuid: "a2c791357c0f445da36dfef65a3155ca" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } } }
I20260929 16:28:40.711755 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:28:40.713066 13830 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e [term 1 FOLLOWER]: Refusing update from remote peer 9148a3dcb23343fca9edc22446665622: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:40.713936 13963 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0918733557a1417fb6f604a5bd8bbf0e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:28:40.721828 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9148a3dcb23343fca9edc22446665622. Latest consensus state: current_term: 1 leader_uuid: "9148a3dcb23343fca9edc22446665622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9148a3dcb23343fca9edc22446665622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "0918733557a1417fb6f604a5bd8bbf0e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } } peers { permanent_uuid: "a2c791357c0f445da36dfef65a3155ca" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } } }
I20260929 16:28:40.722028 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:28:40.728760 13988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9148a3dcb23343fca9edc22446665622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9148a3dcb23343fca9edc22446665622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "0918733557a1417fb6f604a5bd8bbf0e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } } peers { permanent_uuid: "a2c791357c0f445da36dfef65a3155ca" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } } }
I20260929 16:28:40.728971 13988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:40.730676 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9148a3dcb23343fca9edc22446665622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9148a3dcb23343fca9edc22446665622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "0918733557a1417fb6f604a5bd8bbf0e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } } peers { permanent_uuid: "a2c791357c0f445da36dfef65a3155ca" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } } }
I20260929 16:28:40.730856 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:28:40.731371 13991 mvcc.cc:205] Tried to move back new op lower bound from 7334704417588207616 to 7334704417548304384. Current Snapshot: MvccSnapshot[applied={T|T < 7334704417588207616}]
I20260929 16:28:40.734181 13994 mvcc.cc:205] Tried to move back new op lower bound from 7334704417588207616 to 7334704417548304384. Current Snapshot: MvccSnapshot[applied={T|T < 7334704417588207616}]
I20260929 16:28:40.737095 13963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9148a3dcb23343fca9edc22446665622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9148a3dcb23343fca9edc22446665622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "0918733557a1417fb6f604a5bd8bbf0e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } } peers { permanent_uuid: "a2c791357c0f445da36dfef65a3155ca" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } } }
I20260929 16:28:40.737317 13963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622 [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:40.738900 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9148a3dcb23343fca9edc22446665622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9148a3dcb23343fca9edc22446665622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36503 } } peers { permanent_uuid: "0918733557a1417fb6f604a5bd8bbf0e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36387 } } peers { permanent_uuid: "a2c791357c0f445da36dfef65a3155ca" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36493 } } }
I20260929 16:28:40.739070 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:28:40.746317 13992 mvcc.cc:205] Tried to move back new op lower bound from 7334704417588207616 to 7334704417548304384. Current Snapshot: MvccSnapshot[applied={T|T < 7334704417588207616}]
I20260929 16:28:40.747568 13990 catalog_manager.cc:1347] Generated new cluster ID: c03bcbd0dedb41a797a7f8f591391fc0
I20260929 16:28:40.747658 13990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:28:40.786834 13990 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:28:40.787985 13990 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:28:40.797849 13990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9148a3dcb23343fca9edc22446665622: Generated new TSK 0
I20260929 16:28:40.798962 13990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:28:41.568493 13951 catalog_manager.cc:1259] Loaded cluster ID: c03bcbd0dedb41a797a7f8f591391fc0
I20260929 16:28:41.568585 13951 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e: loading cluster ID for follower catalog manager: success
I20260929 16:28:41.570395 13951 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e: acquiring CA information for follower catalog manager: success
I20260929 16:28:41.571660 13951 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0918733557a1417fb6f604a5bd8bbf0e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:28:41.587098 13985 catalog_manager.cc:1259] Loaded cluster ID: c03bcbd0dedb41a797a7f8f591391fc0
I20260929 16:28:41.587188 13985 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca: loading cluster ID for follower catalog manager: success
I20260929 16:28:41.588991 13985 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca: acquiring CA information for follower catalog manager: success
I20260929 16:28:41.590284 13985 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a2c791357c0f445da36dfef65a3155ca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:28:41.859282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 13715
I20260929 16:28:41.866065 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 13790
I20260929 16:28:41.872699 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 13864
2026-09-29T16:28:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3261 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:28:41 dist-test-slave-vq9v krb5kdc[14005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:28:41 dist-test-slave-vq9v krb5kdc[14005](info): set up 2 sockets
Sep 29 16:28:41 dist-test-slave-vq9v krb5kdc[14005](info): commencing operation
krb5kdc: starting...
W20260929 16:28:42.925421 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:28:43 dist-test-slave-vq9v krb5kdc[14005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:28:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:28:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:28:43.052699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42819
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42819
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:43.169353 14018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:43.169690 14018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:43.169775 14018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:43.174006 14018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:28:43.174119 14018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:43.174160 14018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:43.174199 14018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:28:43.174235 14018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:28:43.179766 14018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:43.180900 14018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:43.182224 14018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:28:43.192096 14018 server_base.cc:1034] running on GCE node
W20260929 16:28:43.192001 14023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:43.192142 14026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:43.192020 14024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:43.192901 14018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:43.194146 14018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:43.195322 14018 hybrid_clock.cc:648] HybridClock initialized: now 1790699323195308 us; error 26 us; skew 500 ppm
Sep 29 16:28:43 dist-test-slave-vq9v krb5kdc[14005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:43.198402 14018 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:28:43.199378 14018 webserver.cc:460] Webserver started at http://127.0.98.62:42763/ using document root <none> and password file <none>
I20260929 16:28:43.199998 14018 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:43.200095 14018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:43.200328 14018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:43.202471 14018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "98379fb9676b4da1bbfa1bc41539e8c2"
format_stamp: "Formatted at 2026-09-29 16:28:43 on dist-test-slave-vq9v"
server_key: "b87f6301252488de98fcc557d5652ae2"
server_key_iv: "8ac864afcc0020106f1f660b7096eb06"
server_key_version: "encryptionkey@0"
I20260929 16:28:43.203060 14018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "98379fb9676b4da1bbfa1bc41539e8c2"
format_stamp: "Formatted at 2026-09-29 16:28:43 on dist-test-slave-vq9v"
server_key: "b87f6301252488de98fcc557d5652ae2"
server_key_iv: "8ac864afcc0020106f1f660b7096eb06"
server_key_version: "encryptionkey@0"
I20260929 16:28:43.207003 14018 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:28:43.209684 14033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:43.210963 14018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:28:43.211220 14018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "98379fb9676b4da1bbfa1bc41539e8c2"
format_stamp: "Formatted at 2026-09-29 16:28:43 on dist-test-slave-vq9v"
server_key: "b87f6301252488de98fcc557d5652ae2"
server_key_iv: "8ac864afcc0020106f1f660b7096eb06"
server_key_version: "encryptionkey@0"
I20260929 16:28:43.211364 14018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:43.248760 14018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:43.252400 14018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:43.252627 14018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:43.260421 14018 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42819
I20260929 16:28:43.260500 14085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42819 every 8 connection(s)
I20260929 16:28:43.261495 14018 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:28:43.264345 14086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:43.270109 14086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2: Bootstrap starting.
I20260929 16:28:43.270159 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 14018
I20260929 16:28:43.270280 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:28:43.270570 392 external_mini_cluster.cc:1428] Setting key 9255492b0f0ea2f4b2d6ef7dff4f00c8
I20260929 16:28:43.272840 14086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:43.273697 14086 log.cc:824] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:43.275799 14086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2: No bootstrap required, opened a new log
Sep 29 16:28:43 dist-test-slave-vq9v krb5kdc[14005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:28:43.280582 14086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98379fb9676b4da1bbfa1bc41539e8c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42819 } }
I20260929 16:28:43.280856 14086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:43.280937 14086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98379fb9676b4da1bbfa1bc41539e8c2, State: Initialized, Role: FOLLOWER
I20260929 16:28:43.281543 14086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98379fb9676b4da1bbfa1bc41539e8c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42819 } }
I20260929 16:28:43.281721 14086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:28:43.281829 14086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:28:43.281957 14086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:43.282924 14086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98379fb9676b4da1bbfa1bc41539e8c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42819 } }
I20260929 16:28:43.283293 14086 leader_election.cc:302] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98379fb9676b4da1bbfa1bc41539e8c2; no voters:
I20260929 16:28:43.283645 14086 leader_election.cc:288] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:28:43.283797 14091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:28:43.284055 14091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2 [term 1 LEADER]: Becoming Leader. State: Replica: 98379fb9676b4da1bbfa1bc41539e8c2, State: Running, Role: LEADER
I20260929 16:28:43.284422 14091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98379fb9676b4da1bbfa1bc41539e8c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42819 } }
I20260929 16:28:43.285130 14086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:28:43.286118 14092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98379fb9676b4da1bbfa1bc41539e8c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98379fb9676b4da1bbfa1bc41539e8c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42819 } } }
I20260929 16:28:43.286250 14092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:43.286651 14093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98379fb9676b4da1bbfa1bc41539e8c2. Latest consensus state: current_term: 1 leader_uuid: "98379fb9676b4da1bbfa1bc41539e8c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98379fb9676b4da1bbfa1bc41539e8c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42819 } } }
I20260929 16:28:43.286729 14093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:43.287119 14089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:43.271977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49106 (local address 127.0.98.62:42819)
0929 16:28:43.272424 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:43.272439 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:43.272477 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:28:43.273192 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:43.273263 (+ 71us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:43.273612 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:43.273942 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:43.275079 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:43.276102 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:43.277272 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:43.277732 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:43.277852 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:43.280738 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:43.280742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:43.280756 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:43.280784 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:43.282856 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:43.283375 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:43.283381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:43.283388 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:43.283572 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:43.283987 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:43.283991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:43.283993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:43.284506 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:43.285400 (+ 894us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:43.285564 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 16:28:43.285890 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":142,"threads_started":1}
I20260929 16:28:43.290335 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 16:28:43.290573 14106 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:28:43.290668 14106 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:28:43.290766 14100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:28:43.291679 14100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:28:43.297436 14100 catalog_manager.cc:1347] Generated new cluster ID: 5bebc51c5f0645e2b4656563f06b2942
I20260929 16:28:43.297514 14100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:28:43.311084 14100 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:28:43.312137 14100 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:28:43.323961 14100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98379fb9676b4da1bbfa1bc41539e8c2: Generated new TSK 0
I20260929 16:28:43.324748 14100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:28:43.346395 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 14018
2026-09-29T16:28:43Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1470 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:28:43 dist-test-slave-vq9v krb5kdc[14113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:28:43 dist-test-slave-vq9v krb5kdc[14113](info): set up 2 sockets
Sep 29 16:28:43 dist-test-slave-vq9v krb5kdc[14113](info): commencing operation
krb5kdc: starting...
W20260929 16:28:44.441655 392 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:28:44 dist-test-slave-vq9v krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:28:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:28:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:28:44.573673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39305
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:44.685117 14126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:44.685477 14126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:44.685557 14126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:44.689656 14126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:28:44.689744 14126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:44.689783 14126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:44.689843 14126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:28:44.689880 14126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:28:44.695343 14126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:44.696509 14126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:44.697618 14126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:44.707329 14134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:44.707329 14132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:44.707404 14131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:44.707934 14126 server_base.cc:1034] running on GCE node
I20260929 16:28:44.708570 14126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:44.709995 14126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:44.711252 14126 hybrid_clock.cc:648] HybridClock initialized: now 1790699324711230 us; error 39 us; skew 500 ppm
Sep 29 16:28:44 dist-test-slave-vq9v krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:44.720106 14126 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:28:44.721174 14126 webserver.cc:460] Webserver started at http://127.0.98.62:41621/ using document root <none> and password file <none>
I20260929 16:28:44.721884 14126 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:44.721979 14126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:44.722186 14126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:44.724406 14126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3"
format_stamp: "Formatted at 2026-09-29 16:28:44 on dist-test-slave-vq9v"
server_key: "735567f8adcd83831254c7e8e7e09893"
server_key_iv: "e63080736585eb95bcc662142d2e30eb"
server_key_version: "encryptionkey@0"
I20260929 16:28:44.724972 14126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3"
format_stamp: "Formatted at 2026-09-29 16:28:44 on dist-test-slave-vq9v"
server_key: "735567f8adcd83831254c7e8e7e09893"
server_key_iv: "e63080736585eb95bcc662142d2e30eb"
server_key_version: "encryptionkey@0"
I20260929 16:28:44.729472 14126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:28:44.732259 14141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:44.734112 14126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:28:44.734265 14126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3"
format_stamp: "Formatted at 2026-09-29 16:28:44 on dist-test-slave-vq9v"
server_key: "735567f8adcd83831254c7e8e7e09893"
server_key_iv: "e63080736585eb95bcc662142d2e30eb"
server_key_version: "encryptionkey@0"
I20260929 16:28:44.734467 14126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:44.757795 14126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:44.761840 14126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:44.762109 14126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:44.770272 14126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39305
I20260929 16:28:44.770710 14193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39305 every 8 connection(s)
I20260929 16:28:44.771466 14126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:28:44.774680 14194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:44.780776 14194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3: Bootstrap starting.
I20260929 16:28:44.780993 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 14126
I20260929 16:28:44.781131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:28:44.781488 392 external_mini_cluster.cc:1428] Setting key 597f4dd287e7a9a9387eedc2cdcab2b9
I20260929 16:28:44.783833 14194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:44.784650 14194 log.cc:824] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:44.786814 14194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3: No bootstrap required, opened a new log
Sep 29 16:28:44 dist-test-slave-vq9v krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:28:44.790796 14194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } }
I20260929 16:28:44.791080 14194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:44.791167 14194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b9ba30cc08a4dad9ea3b0b818094dc3, State: Initialized, Role: FOLLOWER
I20260929 16:28:44.791770 14194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } }
I20260929 16:28:44.791935 14194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:28:44.792034 14194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:28:44.792160 14194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:44.793644 14194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } }
I20260929 16:28:44.794085 14194 leader_election.cc:302] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b9ba30cc08a4dad9ea3b0b818094dc3; no voters:
I20260929 16:28:44.794545 14194 leader_election.cc:288] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:28:44.794668 14199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:28:44.794979 14199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 1 LEADER]: Becoming Leader. State: Replica: 0b9ba30cc08a4dad9ea3b0b818094dc3, State: Running, Role: LEADER
I20260929 16:28:44.795406 14199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } }
I20260929 16:28:44.796025 14194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:28:44.796666 14199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b9ba30cc08a4dad9ea3b0b818094dc3. Latest consensus state: current_term: 1 leader_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } } }
I20260929 16:28:44.796779 14199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:44.797175 14200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } } }
I20260929 16:28:44.797366 14200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:44.797649 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:44.782980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41674 (local address 127.0.98.62:39305)
0929 16:28:44.783485 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:44.783500 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:44.783535 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:28:44.784377 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:44.784398 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:44.784785 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:44.785151 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:44.786021 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:44.787075 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:44.787977 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:44.788502 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:44.788685 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:44.791353 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:44.791360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:44.791380 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:44.791419 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:44.793930 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:44.794485 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:44.794495 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:44.794503 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:44.794672 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:44.795115 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:44.795119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:44.795121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:44.795669 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:44.795891 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:44.796242 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 16:28:44.796572 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":153,"threads_started":1}
W20260929 16:28:44.801719 14214 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:28:44.801796 14214 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:28:44.801884 14203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:28:44.802793 14203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:28:44.808753 14203 catalog_manager.cc:1347] Generated new cluster ID: 912a670d935a45d1af400c0f82fc1ef2
I20260929 16:28:44.808848 14203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:28:44.828666 14203 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:28:44.830016 14203 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:28:44.840778 14203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3: Generated new TSK 0
I20260929 16:28:44.841734 14203 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 16:28:44.881819 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39305
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:44.997077 14220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:44.997457 14220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:44.997546 14220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:45.001755 14220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:45.001870 14220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:45.001969 14220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:28:45.007593 14220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:45.008780 14220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:45.009955 14220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:45.020815 14226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:45.020841 14225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:45.020803 14228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:45.020906 14220 server_base.cc:1034] running on GCE node
I20260929 16:28:45.021554 14220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:45.022292 14220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:45.023483 14220 hybrid_clock.cc:648] HybridClock initialized: now 1790699325023473 us; error 30 us; skew 500 ppm
Sep 29 16:28:45 dist-test-slave-vq9v krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:45.026453 14220 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:28:45.027390 14220 webserver.cc:460] Webserver started at http://127.0.98.1:37177/ using document root <none> and password file <none>
I20260929 16:28:45.028029 14220 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:45.028127 14220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:45.028357 14220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:45.030522 14220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "9f4683766c12485f873f8103679e5ff5"
format_stamp: "Formatted at 2026-09-29 16:28:45 on dist-test-slave-vq9v"
server_key: "ef58146c263e36f2c8f7accb76fd5051"
server_key_iv: "d77357fd425afd115983d6f908879908"
server_key_version: "encryptionkey@0"
I20260929 16:28:45.031138 14220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9f4683766c12485f873f8103679e5ff5"
format_stamp: "Formatted at 2026-09-29 16:28:45 on dist-test-slave-vq9v"
server_key: "ef58146c263e36f2c8f7accb76fd5051"
server_key_iv: "d77357fd425afd115983d6f908879908"
server_key_version: "encryptionkey@0"
I20260929 16:28:45.035233 14220 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 16:28:45.038115 14235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:45.039768 14220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:28:45.039917 14220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "9f4683766c12485f873f8103679e5ff5"
format_stamp: "Formatted at 2026-09-29 16:28:45 on dist-test-slave-vq9v"
server_key: "ef58146c263e36f2c8f7accb76fd5051"
server_key_iv: "d77357fd425afd115983d6f908879908"
server_key_version: "encryptionkey@0"
I20260929 16:28:45.040091 14220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:45.051120 14220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:45.054471 14220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:45.054692 14220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:45.055305 14220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:28:45.056270 14220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:45.056339 14220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:45.056411 14220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:45.056452 14220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:45.066478 14220 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35989
I20260929 16:28:45.066588 14348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35989 every 8 connection(s)
I20260929 16:28:45.067498 14220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:28:45.068924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 14220
I20260929 16:28:45.069092 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:28:45.069460 392 external_mini_cluster.cc:1428] Setting key c5723e460c141cd8e2dd86e15cd77a7b
Sep 29 16:28:45 dist-test-slave-vq9v krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 16:28:45.083425 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.069655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37019 (local address 127.0.98.62:39305)
0929 16:28:45.069807 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:45.069813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:45.070867 (+ 1054us) server_negotiation.cc:407] Connection header received
0929 16:28:45.072412 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:45.072417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:45.072478 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:45.072561 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:45.073947 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.074493 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.075791 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.076055 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.076170 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.079096 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:45.079099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:45.079101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:45.079131 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:45.080790 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:45.081529 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:45.081532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:45.081533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:45.081624 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:45.082093 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:45.082096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:45.082098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:45.082313 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:45.082430 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:45.083134 (+ 704us) server_negotiation.cc:299] Negotiation successful
0929 16:28:45.083264 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:28:45.084717 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.070125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.1:37019)
0929 16:28:45.070688 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:45.070736 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:45.072157 (+ 1421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:45.072756 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:45.072764 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:45.073157 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:45.073767 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:45.073783 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.074635 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.074638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:45.075648 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:45.075658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.076230 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.076234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:45.076426 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.077079 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:45.077096 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:45.078883 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:45.080916 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:45.080923 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:45.080938 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:45.081388 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:45.081770 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:45.081773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:45.081775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:45.081970 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:45.082432 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:45.082443 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:45.082800 (+ 357us) client_negotiation.cc:769] Sending connection context
0929 16:28:45.083048 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 16:28:45.083547 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":199,"threads_started":1}
I20260929 16:28:45.086324 14349 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39305
I20260929 16:28:45.086653 14349 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:45.087261 14349 heartbeater.cc:511] Master 127.0.98.62:39305 requested a full tablet report, sending...
I20260929 16:28:45.089107 14158 ts_manager.cc:194] Registered new tserver with Master: 9f4683766c12485f873f8103679e5ff5 (127.0.98.1:35989)
I20260929 16:28:45.090399 14158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 16:28:45.101394 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39305
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:45.220116 14354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:45.220415 14354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:45.220461 14354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:45.224730 14354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:45.224817 14354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:45.224879 14354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:28:45.230600 14354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:45.231690 14354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:45.232752 14354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:45.243268 14359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:45.243377 14354 server_base.cc:1034] running on GCE node
W20260929 16:28:45.243278 14362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:45.243598 14360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:45.244047 14354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:45.244854 14354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:45.246058 14354 hybrid_clock.cc:648] HybridClock initialized: now 1790699325246014 us; error 56 us; skew 500 ppm
Sep 29 16:28:45 dist-test-slave-vq9v krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:45.249579 14354 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:28:45.250579 14354 webserver.cc:460] Webserver started at http://127.0.98.2:43893/ using document root <none> and password file <none>
I20260929 16:28:45.251145 14354 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:45.251207 14354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:45.251380 14354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:45.253635 14354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "738f451eb9654817824e91c33026f856"
format_stamp: "Formatted at 2026-09-29 16:28:45 on dist-test-slave-vq9v"
server_key: "c8bc4434c993c2186ee4952446bf1e4c"
server_key_iv: "5a7c00c78d26ac3d5d2f0fc5efb7d23a"
server_key_version: "encryptionkey@0"
I20260929 16:28:45.254184 14354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "738f451eb9654817824e91c33026f856"
format_stamp: "Formatted at 2026-09-29 16:28:45 on dist-test-slave-vq9v"
server_key: "c8bc4434c993c2186ee4952446bf1e4c"
server_key_iv: "5a7c00c78d26ac3d5d2f0fc5efb7d23a"
server_key_version: "encryptionkey@0"
I20260929 16:28:45.259004 14354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:28:45.262197 14369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:45.263638 14354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:28:45.263782 14354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "738f451eb9654817824e91c33026f856"
format_stamp: "Formatted at 2026-09-29 16:28:45 on dist-test-slave-vq9v"
server_key: "c8bc4434c993c2186ee4952446bf1e4c"
server_key_iv: "5a7c00c78d26ac3d5d2f0fc5efb7d23a"
server_key_version: "encryptionkey@0"
I20260929 16:28:45.263912 14354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:45.278782 14354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:45.282393 14354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:45.282610 14354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:45.283242 14354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:28:45.284101 14354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:45.284153 14354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:45.284229 14354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:45.284267 14354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:45.294164 14354 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40959
I20260929 16:28:45.294184 14482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40959 every 8 connection(s)
I20260929 16:28:45.295164 14354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:28:45.298625 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 14354
I20260929 16:28:45.298749 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:28:45.299041 392 external_mini_cluster.cc:1428] Setting key e2966e1ee3b9e83244cebf0e6c953466
Sep 29 16:28:45 dist-test-slave-vq9v krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 16:28:45.312475 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.297035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51559 (local address 127.0.98.62:39305)
0929 16:28:45.297191 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:45.297229 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:45.299012 (+ 1783us) server_negotiation.cc:407] Connection header received
0929 16:28:45.300184 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:45.300187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:45.300239 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:45.300330 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:45.301791 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.302496 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.304469 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.304744 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.304868 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.308295 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:45.308298 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:45.308300 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:45.308330 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:45.310141 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:45.310763 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:45.310769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:45.310775 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:45.310859 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:45.311249 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:45.311252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:45.311254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:45.311463 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:45.311582 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:45.312151 (+ 569us) server_negotiation.cc:299] Negotiation successful
0929 16:28:45.312301 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:28:45.313699 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.297640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.2:51559)
0929 16:28:45.298742 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:45.298879 (+ 137us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:45.299961 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:45.300557 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:45.300565 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:45.301027 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:45.301633 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:45.301648 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.302660 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.302665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:45.304305 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:45.304323 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.304959 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.304967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:45.305242 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.306006 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:45.306034 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:45.308062 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:45.310301 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:45.310308 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:45.310328 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:45.310637 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:45.310961 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:45.310964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:45.310966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:45.311138 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:45.311584 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:45.311594 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:45.311894 (+ 300us) client_negotiation.cc:769] Sending connection context
0929 16:28:45.312098 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 16:28:45.312473 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":558,"threads_started":1}
I20260929 16:28:45.315445 14483 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39305
I20260929 16:28:45.315734 14483 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:45.316298 14483 heartbeater.cc:511] Master 127.0.98.62:39305 requested a full tablet report, sending...
I20260929 16:28:45.317610 14158 ts_manager.cc:194] Registered new tserver with Master: 738f451eb9654817824e91c33026f856 (127.0.98.2:40959)
I20260929 16:28:45.318216 14158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 16:28:45.332293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39305
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:45.456521 14488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:45.456919 14488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:45.457017 14488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:45.461973 14488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:45.462090 14488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:45.462200 14488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:28:45.468333 14488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:45.469641 14488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:45.470717 14488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:45.481390 14493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:45.481376 14494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:45.481451 14488 server_base.cc:1034] running on GCE node
W20260929 16:28:45.481376 14496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:45.482057 14488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:45.482769 14488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:45.483938 14488 hybrid_clock.cc:648] HybridClock initialized: now 1790699325483920 us; error 38 us; skew 500 ppm
Sep 29 16:28:45 dist-test-slave-vq9v krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:45.486985 14488 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:28:45.487967 14488 webserver.cc:460] Webserver started at http://127.0.98.3:40839/ using document root <none> and password file <none>
I20260929 16:28:45.488628 14488 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:45.488727 14488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:45.488971 14488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:28:45.491266 14488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "58e3238aaeea4f328a9eb827497df008"
format_stamp: "Formatted at 2026-09-29 16:28:45 on dist-test-slave-vq9v"
server_key: "84cb7047170beafa67f580be09ad4680"
server_key_iv: "0d0d570a89d060bb4816dbaae7434859"
server_key_version: "encryptionkey@0"
I20260929 16:28:45.491940 14488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "58e3238aaeea4f328a9eb827497df008"
format_stamp: "Formatted at 2026-09-29 16:28:45 on dist-test-slave-vq9v"
server_key: "84cb7047170beafa67f580be09ad4680"
server_key_iv: "0d0d570a89d060bb4816dbaae7434859"
server_key_version: "encryptionkey@0"
I20260929 16:28:45.496554 14488 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:28:45.500118 14503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:45.501909 14488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:28:45.502101 14488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "58e3238aaeea4f328a9eb827497df008"
format_stamp: "Formatted at 2026-09-29 16:28:45 on dist-test-slave-vq9v"
server_key: "84cb7047170beafa67f580be09ad4680"
server_key_iv: "0d0d570a89d060bb4816dbaae7434859"
server_key_version: "encryptionkey@0"
I20260929 16:28:45.502249 14488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:45.526523 14488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:45.530366 14488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:45.530613 14488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:45.531319 14488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:28:45.532188 14488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:45.532264 14488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:45.532337 14488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:45.532382 14488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:45.542554 14488 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35689
I20260929 16:28:45.542608 14616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35689 every 8 connection(s)
I20260929 16:28:45.543601 14488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:28:45.549911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 14488
I20260929 16:28:45.550069 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:28:45.550355 392 external_mini_cluster.cc:1428] Setting key aee15a6d3d21c0d04ddfaa9423876caa
Sep 29 16:28:45 dist-test-slave-vq9v krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:28:45.558797 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.545824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46325 (local address 127.0.98.62:39305)
0929 16:28:45.546017 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:45.546021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:45.546931 (+ 910us) server_negotiation.cc:407] Connection header received
0929 16:28:45.548347 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:45.548350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:45.548421 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:45.548544 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:45.549912 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.550594 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.551886 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.552133 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.552263 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.554968 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:45.554971 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:45.554974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:45.555002 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:45.556459 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:45.557022 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:45.557026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:45.557027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:45.557107 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:45.557581 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:45.557583 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:45.557585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:45.557774 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:45.557854 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:45.558533 (+ 679us) server_negotiation.cc:299] Negotiation successful
0929 16:28:45.558669 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:28:45.559584 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.546237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.3:46325)
0929 16:28:45.546759 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:45.546807 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:45.548107 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:45.548736 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:45.548748 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:45.549160 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:45.549751 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:45.549766 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.550722 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.550725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:45.551755 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:45.551765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.552308 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.552316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:45.552511 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.553142 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:45.553160 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:45.554779 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:45.556603 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:45.556608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:45.556623 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:45.556906 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:45.557191 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:45.557193 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:45.557194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:45.557486 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:45.557920 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:45.557928 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:45.558213 (+ 285us) client_negotiation.cc:769] Sending connection context
0929 16:28:45.558401 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 16:28:45.558702 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":134,"threads_started":1}
I20260929 16:28:45.560986 14617 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39305
I20260929 16:28:45.561349 14617 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:45.561942 14617 heartbeater.cc:511] Master 127.0.98.62:39305 requested a full tablet report, sending...
I20260929 16:28:45.563329 14158 ts_manager.cc:194] Registered new tserver with Master: 58e3238aaeea4f328a9eb827497df008 (127.0.98.3:35689)
I20260929 16:28:45.564080 14158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46325
I20260929 16:28:45.564949 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:28:45.575951 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.567334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41678 (local address 127.0.98.62:39305)
0929 16:28:45.567498 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:45.567501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:45.567715 (+ 214us) server_negotiation.cc:407] Connection header received
0929 16:28:45.567804 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:45.567807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:45.567914 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:45.568087 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:45.568967 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.569608 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.570497 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.570734 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.570852 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.572324 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:45.572328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:45.572331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:45.572366 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:45.574025 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:45.574581 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:45.574585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:45.574589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:45.574679 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:45.574967 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:45.574970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:45.574972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:45.575247 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:45.575394 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:45.575639 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 16:28:45.575768 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:28:45.579552 14158 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:28:45.582918 14158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:28:45.599743 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.592835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44854 (local address 127.0.98.1:35989)
0929 16:28:45.593472 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:45.593479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:45.594079 (+ 600us) server_negotiation.cc:407] Connection header received
0929 16:28:45.594313 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:45.594319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:45.594490 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:45.594583 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:45.595076 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.596172 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.597948 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.598880 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.599011 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.599194 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:45.599453 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 16:28:45.599575 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":84,"threads_started":1}
I20260929 16:28:45.599779 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.592702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40959 (local address 127.0.0.1:59562)
0929 16:28:45.593635 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:45.593649 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:45.593765 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:45.594504 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:45.594510 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:45.594532 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:45.594692 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:45.594699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.596228 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.596231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:45.598050 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:45.598062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.599161 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.599165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:45.599250 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.599268 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 16:28:45.599427 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 16:28:45.599614 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":180,"spinlock_wait_cycles":3712,"thread_start_us":130,"threads_started":1}
I20260929 16:28:45.599838 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.593504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59562 (local address 127.0.98.2:40959)
0929 16:28:45.593948 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:45.593961 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:45.593986 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:28:45.594066 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:45.594071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:45.594303 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:45.594503 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:45.594816 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.596085 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.598230 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.599031 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.599135 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.599256 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:45.599571 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 16:28:45.599681 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":97,"threads_started":1}
I20260929 16:28:45.600432 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.592598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35989 (local address 127.0.0.1:44854)
0929 16:28:45.593990 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:45.594004 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:45.594152 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:45.594741 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:45.594744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:45.594771 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:45.594949 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:45.594954 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.596264 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.596270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:45.597810 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:45.597821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.599044 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.599048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:45.599204 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.599263 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 16:28:45.599409 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 16:28:45.600283 (+ 874us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1332,"mutex_wait_us":530,"spinlock_wait_cycles":9472,"thread_start_us":130,"threads_started":1}
I20260929 16:28:45.603478 14417 tablet_service.cc:1467] Processing CreateTablet for tablet 96feb4f0b99b43a9b6c7ab153ed9cb83 (DEFAULT_TABLE table=test-table [id=b8bdc250712e4d138d7be4d362d4f67a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:45.604001 14283 tablet_service.cc:1467] Processing CreateTablet for tablet 96feb4f0b99b43a9b6c7ab153ed9cb83 (DEFAULT_TABLE table=test-table [id=b8bdc250712e4d138d7be4d362d4f67a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:45.604656 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.592598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35689 (local address 127.0.0.1:44602)
0929 16:28:45.593558 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:45.593614 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:45.593765 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:45.594723 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:45.594728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:45.594765 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:45.594947 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:45.594954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.596025 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.596030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:45.598069 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:45.598075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.600679 (+ 2604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.600683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:45.600796 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.600819 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:28:45.600893 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 16:28:45.604480 (+ 3587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":257,"spinlock_wait_cycles":8832,"thread_start_us":84,"threads_started":1}
I20260929 16:28:45.604812 14417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96feb4f0b99b43a9b6c7ab153ed9cb83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:45.605072 14283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96feb4f0b99b43a9b6c7ab153ed9cb83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:45.605283 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.592764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44602 (local address 127.0.98.3:35689)
0929 16:28:45.594082 (+ 1318us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:45.594095 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:45.594114 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:28:45.594188 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:45.594196 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:45.594404 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:45.594554 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:45.595105 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.595890 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.599328 (+ 3438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.600336 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.600454 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.600619 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:45.604485 (+ 3866us) server_negotiation.cc:299] Negotiation successful
0929 16:28:45.604623 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1201,"thread_start_us":116,"threads_started":1}
I20260929 16:28:45.609298 14551 tablet_service.cc:1467] Processing CreateTablet for tablet 96feb4f0b99b43a9b6c7ab153ed9cb83 (DEFAULT_TABLE table=test-table [id=b8bdc250712e4d138d7be4d362d4f67a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:45.612310 14551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96feb4f0b99b43a9b6c7ab153ed9cb83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:45.620726 14634 tablet_bootstrap.cc:492] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856: Bootstrap starting.
I20260929 16:28:45.621712 14635 tablet_bootstrap.cc:492] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 58e3238aaeea4f328a9eb827497df008: Bootstrap starting.
I20260929 16:28:45.624567 14635 tablet_bootstrap.cc:654] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 58e3238aaeea4f328a9eb827497df008: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:45.625782 14634 tablet_bootstrap.cc:654] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:45.626191 14635 log.cc:824] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 58e3238aaeea4f328a9eb827497df008: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:45.626809 14634 log.cc:824] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:45.627087 14636 tablet_bootstrap.cc:492] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5: Bootstrap starting.
I20260929 16:28:45.628821 14635 tablet_bootstrap.cc:492] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 58e3238aaeea4f328a9eb827497df008: No bootstrap required, opened a new log
I20260929 16:28:45.629060 14635 ts_tablet_manager.cc:1397] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 58e3238aaeea4f328a9eb827497df008: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 16:28:45.629151 14634 tablet_bootstrap.cc:492] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856: No bootstrap required, opened a new log
I20260929 16:28:45.629410 14634 ts_tablet_manager.cc:1397] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 16:28:45.633677 14634 raft_consensus.cc:348] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:45.633677 14635 raft_consensus.cc:348] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 58e3238aaeea4f328a9eb827497df008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:45.634040 14635 raft_consensus.cc:374] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 58e3238aaeea4f328a9eb827497df008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:45.634130 14635 raft_consensus.cc:729] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 58e3238aaeea4f328a9eb827497df008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e3238aaeea4f328a9eb827497df008, State: Initialized, Role: FOLLOWER
I20260929 16:28:45.634196 14634 raft_consensus.cc:374] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:45.634295 14634 raft_consensus.cc:729] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 738f451eb9654817824e91c33026f856, State: Initialized, Role: FOLLOWER
I20260929 16:28:45.634776 14634 consensus_queue.cc:260] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:45.634776 14635 consensus_queue.cc:260] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 58e3238aaeea4f328a9eb827497df008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:45.635955 14635 ts_tablet_manager.cc:1428] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 58e3238aaeea4f328a9eb827497df008: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 16:28:45.635955 14634 ts_tablet_manager.cc:1428] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:28:45.636350 14617 heartbeater.cc:503] Master 127.0.98.62:39305 was elected leader, sending a full tablet report...
I20260929 16:28:45.636345 14483 heartbeater.cc:503] Master 127.0.98.62:39305 was elected leader, sending a full tablet report...
I20260929 16:28:45.638227 14636 tablet_bootstrap.cc:654] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:45.639505 14636 log.cc:824] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:45.642503 14636 tablet_bootstrap.cc:492] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5: No bootstrap required, opened a new log
I20260929 16:28:45.642876 14636 ts_tablet_manager.cc:1397] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20260929 16:28:45.647513 14636 raft_consensus.cc:348] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:45.647810 14636 raft_consensus.cc:374] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:45.647853 14636 raft_consensus.cc:729] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f4683766c12485f873f8103679e5ff5, State: Initialized, Role: FOLLOWER
I20260929 16:28:45.648373 14636 consensus_queue.cc:260] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:45.649132 14349 heartbeater.cc:503] Master 127.0.98.62:39305 was elected leader, sending a full tablet report...
I20260929 16:28:45.649509 14636 ts_tablet_manager.cc:1428] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:28:45.660763 14642 raft_consensus.cc:482] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:28:45.661146 14642 raft_consensus.cc:504] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:45.662484 14642 leader_election.cc:288] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58e3238aaeea4f328a9eb827497df008 (127.0.98.3:35689), 738f451eb9654817824e91c33026f856 (127.0.98.2:40959)
I20260929 16:28:45.668066 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.662973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58749 (local address 127.0.98.2:40959)
0929 16:28:45.663206 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:45.663212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:45.663300 (+ 88us) server_negotiation.cc:407] Connection header received
0929 16:28:45.663563 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:45.663567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:45.663634 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:45.663682 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:45.664222 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.665189 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.666851 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.667301 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.667450 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.667534 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:45.667838 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 16:28:45.667923 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260929 16:28:45.668100 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.662749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40959 (local address 127.0.98.1:58749)
0929 16:28:45.663188 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:45.663207 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:45.663406 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:45.663858 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:45.663861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:45.663873 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:45.664087 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:45.664095 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.665348 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.665352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:45.666702 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:45.666718 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.667455 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.667459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:45.667662 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.667697 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:28:45.667816 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 16:28:45.667904 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":55,"spinlock_wait_cycles":6016,"thread_start_us":148,"threads_started":1}
I20260929 16:28:45.669867 14437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96feb4f0b99b43a9b6c7ab153ed9cb83" candidate_uuid: "9f4683766c12485f873f8103679e5ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "738f451eb9654817824e91c33026f856" is_pre_election: true
I20260929 16:28:45.670182 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.662752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35689 (local address 127.0.98.1:44781)
0929 16:28:45.663359 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:45.663372 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:45.663472 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:45.663838 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:45.663844 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:45.663868 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:45.664087 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:45.664095 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.666269 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.666272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:45.667908 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:45.667925 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.669656 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.669659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:45.669758 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.669782 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:28:45.669865 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 16:28:45.670024 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":86,"spinlock_wait_cycles":2304,"thread_start_us":100,"threads_started":1}
I20260929 16:28:45.670246 14437 raft_consensus.cc:2454] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f4683766c12485f873f8103679e5ff5 in term 0.
I20260929 16:28:45.670336 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.663123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44781 (local address 127.0.98.3:35689)
0929 16:28:45.663275 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:45.663281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:45.663458 (+ 177us) server_negotiation.cc:407] Connection header received
0929 16:28:45.663594 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:45.663600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:45.663668 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:45.663771 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:45.665335 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.666127 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.668101 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.668996 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.669102 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.669259 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:45.670020 (+ 761us) server_negotiation.cc:299] Negotiation successful
0929 16:28:45.670145 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:28:45.671155 14237 leader_election.cc:302] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 738f451eb9654817824e91c33026f856, 9f4683766c12485f873f8103679e5ff5; no voters:
I20260929 16:28:45.671620 14642 raft_consensus.cc:2781] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:28:45.671729 14642 raft_consensus.cc:482] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:28:45.671799 14642 raft_consensus.cc:3037] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:45.672128 14571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96feb4f0b99b43a9b6c7ab153ed9cb83" candidate_uuid: "9f4683766c12485f873f8103679e5ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e3238aaeea4f328a9eb827497df008" is_pre_election: true
I20260929 16:28:45.672484 14571 raft_consensus.cc:2454] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 58e3238aaeea4f328a9eb827497df008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f4683766c12485f873f8103679e5ff5 in term 0.
I20260929 16:28:45.672951 14642 raft_consensus.cc:504] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:45.673359 14642 leader_election.cc:288] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 58e3238aaeea4f328a9eb827497df008 (127.0.98.3:35689), 738f451eb9654817824e91c33026f856 (127.0.98.2:40959)
I20260929 16:28:45.674093 14571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96feb4f0b99b43a9b6c7ab153ed9cb83" candidate_uuid: "9f4683766c12485f873f8103679e5ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e3238aaeea4f328a9eb827497df008"
I20260929 16:28:45.674238 14571 raft_consensus.cc:3037] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 58e3238aaeea4f328a9eb827497df008 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:45.675339 14571 raft_consensus.cc:2454] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 58e3238aaeea4f328a9eb827497df008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f4683766c12485f873f8103679e5ff5 in term 1.
I20260929 16:28:45.675657 14437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96feb4f0b99b43a9b6c7ab153ed9cb83" candidate_uuid: "9f4683766c12485f873f8103679e5ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "738f451eb9654817824e91c33026f856"
I20260929 16:28:45.675746 14239 leader_election.cc:302] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58e3238aaeea4f328a9eb827497df008, 9f4683766c12485f873f8103679e5ff5; no voters:
I20260929 16:28:45.675806 14437 raft_consensus.cc:3037] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:45.675976 14642 raft_consensus.cc:2781] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:28:45.676239 14642 raft_consensus.cc:686] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 9f4683766c12485f873f8103679e5ff5, State: Running, Role: LEADER
I20260929 16:28:45.676609 14642 consensus_queue.cc:237] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:45.677244 14437 raft_consensus.cc:2454] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f4683766c12485f873f8103679e5ff5 in term 1.
I20260929 16:28:45.680209 14158 catalog_manager.cc:5495] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f4683766c12485f873f8103679e5ff5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9f4683766c12485f873f8103679e5ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } health_report { overall_health: UNKNOWN } } }
I20260929 16:28:45.704105 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.700285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44870 (local address 127.0.98.1:35989)
0929 16:28:45.700438 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:45.700443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:45.700459 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:28:45.700570 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:45.700572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:45.700631 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:45.700750 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:28:45.701021 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.701625 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.702545 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:45.702779 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:45.702890 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:45.703238 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:28:45.703695 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:28:45.703838 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:45.703882 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 16:28:45.703949 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:28:45.711205 14437 raft_consensus.cc:1261] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856 [term 1 FOLLOWER]: Refusing update from remote peer 9f4683766c12485f873f8103679e5ff5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:45.711205 14571 raft_consensus.cc:1261] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 58e3238aaeea4f328a9eb827497df008 [term 1 FOLLOWER]: Refusing update from remote peer 9f4683766c12485f873f8103679e5ff5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:45.712071 14642 consensus_queue.cc:1035] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:28:45.712352 14645 consensus_queue.cc:1035] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:28:45.725739 14650 mvcc.cc:205] Tried to move back new op lower bound from 7334704438105133056 to 7334704437975134208. Current Snapshot: MvccSnapshot[applied={T|T < 7334704438105133056}]
I20260929 16:28:45.726823 14649 mvcc.cc:205] Tried to move back new op lower bound from 7334704438105133056 to 7334704437975134208. Current Snapshot: MvccSnapshot[applied={T|T < 7334704438105133056}]
I20260929 16:28:45.739264 14157 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:28:45.739614 14157 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:28:45.743489 14157 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3: Sending DeleteTablet for 3 replicas of tablet 96feb4f0b99b43a9b6c7ab153ed9cb83
I20260929 16:28:45.744580 14283 tablet_service.cc:1514] Processing DeleteTablet for tablet 96feb4f0b99b43a9b6c7ab153ed9cb83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44854
I20260929 16:28:45.744575 14551 tablet_service.cc:1514] Processing DeleteTablet for tablet 96feb4f0b99b43a9b6c7ab153ed9cb83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44602
I20260929 16:28:45.745188 14657 tablet_replica.cc:330] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5: stopping tablet replica
I20260929 16:28:45.745509 14658 tablet_replica.cc:330] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 58e3238aaeea4f328a9eb827497df008: stopping tablet replica
I20260929 16:28:45.745805 14417 tablet_service.cc:1514] Processing DeleteTablet for tablet 96feb4f0b99b43a9b6c7ab153ed9cb83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59562
I20260929 16:28:45.745931 14658 raft_consensus.cc:2229] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 58e3238aaeea4f328a9eb827497df008 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:28:45.746282 14658 raft_consensus.cc:2258] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 58e3238aaeea4f328a9eb827497df008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:45.746799 14657 raft_consensus.cc:2229] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:28:45.747182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 14220
I20260929 16:28:45.750597 14658 ts_tablet_manager.cc:1905] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 58e3238aaeea4f328a9eb827497df008: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 16:28:45.753866 14142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35989 (error 108)
I20260929 16:28:45.754096 14659 tablet_replica.cc:330] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856: stopping tablet replica
I20260929 16:28:45.754582 14659 raft_consensus.cc:2229] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:28:45.754674 14658 ts_tablet_manager.cc:1918] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 58e3238aaeea4f328a9eb827497df008: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:28:45.754792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 14354
I20260929 16:28:45.754894 14658 log.cc:1196] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 58e3238aaeea4f328a9eb827497df008: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/96feb4f0b99b43a9b6c7ab153ed9cb83
I20260929 16:28:45.755100 14659 raft_consensus.cc:2258] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:45.755308 14658 ts_tablet_manager.cc:1939] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 58e3238aaeea4f328a9eb827497df008: Deleting consensus metadata
I20260929 16:28:45.756395 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.755995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35989 (local address 127.0.0.1:44872)
0929 16:28:45.756213 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:45.756294 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260929 16:28:45.756664 14142 catalog_manager.cc:4584] TS 9f4683766c12485f873f8103679e5ff5 (127.0.98.1:35989): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 96feb4f0b99b43a9b6c7ab153ed9cb83: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35989: connect: Connection refused (error 111)
I20260929 16:28:45.757031 14145 catalog_manager.cc:4841] TS 58e3238aaeea4f328a9eb827497df008 (127.0.98.3:35689): tablet 96feb4f0b99b43a9b6c7ab153ed9cb83 (table test-table [id=b8bdc250712e4d138d7be4d362d4f67a]) successfully deleted
I20260929 16:28:45.762305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 14488
I20260929 16:28:45.764279 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:45.763815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40959 (local address 127.0.0.1:59564)
0929 16:28:45.764079 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:45.764161 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260929 16:28:45.768836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 14126
I20260929 16:28:45.775812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39305
--webserver_interface=127.0.98.62
--webserver_port=41621
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:45.894234 14661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:45.894551 14661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:45.894595 14661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:45.898783 14661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:28:45.898864 14661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:45.898883 14661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:45.898900 14661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:28:45.898916 14661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:28:45.904490 14661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:45.905696 14661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:45.906735 14661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:28:45.920812 14661 server_base.cc:1034] running on GCE node
W20260929 16:28:45.920825 14670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:45.920784 14668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:45.920784 14667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:45.921965 14661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:45.923353 14661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:45.924625 14661 hybrid_clock.cc:648] HybridClock initialized: now 1790699325924604 us; error 39 us; skew 500 ppm
Sep 29 16:28:45 dist-test-slave-vq9v krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:45.927932 14661 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:28:45.929036 14661 webserver.cc:460] Webserver started at http://127.0.98.62:41621/ using document root <none> and password file <none>
I20260929 16:28:45.929762 14661 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:45.929893 14661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:45.934389 14661 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:28:45.936859 14677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:45.938480 14661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:28:45.938657 14661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3"
format_stamp: "Formatted at 2026-09-29 16:28:44 on dist-test-slave-vq9v"
server_key: "735567f8adcd83831254c7e8e7e09893"
server_key_iv: "e63080736585eb95bcc662142d2e30eb"
server_key_version: "encryptionkey@0"
I20260929 16:28:45.939368 14661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:45.956058 14661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:45.960119 14661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:45.960381 14661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:45.968868 14661 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39305
I20260929 16:28:45.969189 14729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39305 every 8 connection(s)
I20260929 16:28:45.970206 14661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:28:45.973397 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 14661
I20260929 16:28:45.973927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35989
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39305
--webserver_port=37177
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:28:45.975605 14730 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:28:45.977523 14730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3: Bootstrap starting.
I20260929 16:28:45.995113 14730 log.cc:824] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:46.004184 14730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:28:46.004694 14730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3: Bootstrap complete.
I20260929 16:28:46.009125 14730 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } }
I20260929 16:28:46.009631 14730 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b9ba30cc08a4dad9ea3b0b818094dc3, State: Initialized, Role: FOLLOWER
I20260929 16:28:46.010202 14730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } }
I20260929 16:28:46.010334 14730 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:28:46.010428 14730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:28:46.010542 14730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:28:46.012583 14730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } }
I20260929 16:28:46.012950 14730 leader_election.cc:302] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b9ba30cc08a4dad9ea3b0b818094dc3; no voters:
I20260929 16:28:46.013352 14730 leader_election.cc:288] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:28:46.013440 14734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:28:46.013701 14734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 2 LEADER]: Becoming Leader. State: Replica: 0b9ba30cc08a4dad9ea3b0b818094dc3, State: Running, Role: LEADER
I20260929 16:28:46.014101 14734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } }
I20260929 16:28:46.014860 14730 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:28:46.015233 14734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } } }
I20260929 16:28:46.015362 14734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:46.015771 14735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b9ba30cc08a4dad9ea3b0b818094dc3. Latest consensus state: current_term: 2 leader_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } } }
I20260929 16:28:46.015966 14735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:46.020568 14750 catalog_manager.cc:1259] Loaded cluster ID: 912a670d935a45d1af400c0f82fc1ef2
I20260929 16:28:46.020653 14750 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3: loading cluster ID for follower catalog manager: success
W20260929 16:28:46.021780 14750 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260929 16:28:46.021881 14750 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260929 16:28:46.021960 14738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:28:46.024734 14738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:28:46.025275 14738 catalog_manager.cc:1259] Loaded cluster ID: 912a670d935a45d1af400c0f82fc1ef2
I20260929 16:28:46.025336 14738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:28:46.026340 14738 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 16:28:46.030701 14738 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 16:28:46.036568 14738 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:28:46.037573 14738 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3: Loaded TSK: 0
I20260929 16:28:46.038888 14738 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:28:46.101493 14732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:46.101811 14732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:46.101889 14732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:46.106190 14732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:46.106297 14732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:46.106400 14732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:28:46.111980 14732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37177
--tserver_master_addrs=127.0.98.62:39305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:46.113107 14732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:46.114251 14732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:46.124564 14760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:46.124697 14757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:46.124588 14758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:46.125157 14732 server_base.cc:1034] running on GCE node
I20260929 16:28:46.125603 14732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:46.126277 14732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:46.127470 14732 hybrid_clock.cc:648] HybridClock initialized: now 1790699326127437 us; error 51 us; skew 500 ppm
Sep 29 16:28:46 dist-test-slave-vq9v krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:46.130666 14732 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:28:46.131767 14732 webserver.cc:460] Webserver started at http://127.0.98.1:37177/ using document root <none> and password file <none>
I20260929 16:28:46.132371 14732 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:46.132484 14732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:46.136960 14732 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:28:46.139485 14767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:46.140880 14732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:28:46.141013 14732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "9f4683766c12485f873f8103679e5ff5"
format_stamp: "Formatted at 2026-09-29 16:28:45 on dist-test-slave-vq9v"
server_key: "ef58146c263e36f2c8f7accb76fd5051"
server_key_iv: "d77357fd425afd115983d6f908879908"
server_key_version: "encryptionkey@0"
I20260929 16:28:46.141657 14732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:46.161026 14732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:46.164673 14732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:46.164906 14732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:46.165678 14732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:28:46.166947 14774 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 16:28:46.171329 14732 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 16:28:46.171417 14732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260929 16:28:46.171473 14732 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 16:28:46.173213 14732 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 16:28:46.173305 14732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260929 16:28:46.173601 14774 tablet_bootstrap.cc:492] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5: Bootstrap starting.
I20260929 16:28:46.184830 14732 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35989
I20260929 16:28:46.184850 14881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35989 every 8 connection(s)
I20260929 16:28:46.186267 14732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:28:46.191948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 14732
I20260929 16:28:46.192466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40959
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39305
--webserver_port=43893
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:28:46.196007 14774 log.cc:824] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:28:46 dist-test-slave-vq9v krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:28:46.204978 14774 tablet_bootstrap.cc:492] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:28:46.205488 14774 tablet_bootstrap.cc:492] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5: Bootstrap complete.
I20260929 16:28:46.207023 14774 ts_tablet_manager.cc:1397] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5: Time spent bootstrapping tablet: real 0.034s user 0.021s sys 0.011s
I20260929 16:28:46.208565 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.188722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.1:41469)
0929 16:28:46.189396 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:46.189451 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:46.190301 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:46.191940 (+ 1639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:46.191949 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:46.192515 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:46.193338 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.193367 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.195535 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.195541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.197137 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.197154 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.198317 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.198322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.198524 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.199258 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:46.199280 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:46.201642 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:46.204502 (+ 2860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:46.204508 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:46.204522 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:46.204890 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:46.205480 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:46.205486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:46.205489 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:46.205712 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:46.206501 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:46.206510 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:46.206875 (+ 365us) client_negotiation.cc:769] Sending connection context
0929 16:28:46.207083 (+ 208us) client_negotiation.cc:240] Negotiation successful
0929 16:28:46.207484 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":154,"threads_started":1}
I20260929 16:28:46.209303 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.188793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41469 (local address 127.0.98.62:39305)
0929 16:28:46.189298 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:46.189319 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:46.189582 (+ 263us) server_negotiation.cc:407] Connection header received
0929 16:28:46.190792 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:46.190808 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:46.191234 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:46.191728 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:46.193587 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.195313 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.197343 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.198136 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.198341 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.201918 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:46.201926 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:46.201943 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:46.201984 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:46.204332 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:46.205099 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:46.205107 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:46.205114 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:46.205320 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:46.205837 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:46.205843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:46.205846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:46.206343 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:46.206492 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:46.207400 (+ 908us) server_negotiation.cc:299] Negotiation successful
0929 16:28:46.207846 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":143,"threads_started":1}
I20260929 16:28:46.210206 14774 raft_consensus.cc:348] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:46.210642 14774 raft_consensus.cc:729] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f4683766c12485f873f8103679e5ff5, State: Initialized, Role: FOLLOWER
I20260929 16:28:46.212474 14882 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39305
I20260929 16:28:46.212735 14882 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:46.212839 14774 consensus_queue.cc:260] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:46.213447 14882 heartbeater.cc:511] Master 127.0.98.62:39305 requested a full tablet report, sending...
I20260929 16:28:46.214128 14774 ts_tablet_manager.cc:1428] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260929 16:28:46.216904 14693 ts_manager.cc:194] Registered new tserver with Master: 9f4683766c12485f873f8103679e5ff5 (127.0.98.1:35989)
I20260929 16:28:46.220592 14693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41469
I20260929 16:28:46.223158 14882 heartbeater.cc:503] Master 127.0.98.62:39305 was elected leader, sending a full tablet report...
Sep 29 16:28:46 dist-test-slave-vq9v krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 16:28:46.233420 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.220019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35989 (local address 127.0.0.1:45034)
0929 16:28:46.220457 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:46.220504 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:46.220645 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:46.221627 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:46.221633 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:46.222078 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:46.222207 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.222215 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.223480 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.223485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.224434 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.224446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.225242 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.225261 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.225402 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.226233 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:46.226251 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:46.228230 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:46.231415 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:46.231425 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:46.231430 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:46.231707 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:46.232043 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:46.232046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:46.232047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:46.232144 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:46.232707 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:46.232712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:46.232854 (+ 142us) client_negotiation.cc:769] Sending connection context
0929 16:28:46.232994 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 16:28:46.233174 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":115,"threads_started":1}
I20260929 16:28:46.233527 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.220113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45034 (local address 127.0.98.1:35989)
0929 16:28:46.220937 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:46.220949 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:46.220974 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:28:46.221052 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:46.221057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:46.221306 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:46.221442 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:46.222355 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.223339 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.224579 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.225112 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.225240 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.228526 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:46.228539 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:46.228588 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:46.228619 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:46.231268 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:46.231833 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:46.231839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:46.231841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:46.231942 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:46.232234 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:46.232237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:46.232239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:46.232572 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:46.232706 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:46.233133 (+ 427us) server_negotiation.cc:299] Negotiation successful
0929 16:28:46.233354 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":111,"threads_started":1}
I20260929 16:28:46.234800 14816 tablet_service.cc:1514] Processing DeleteTablet for tablet 96feb4f0b99b43a9b6c7ab153ed9cb83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45034
I20260929 16:28:46.235929 14891 tablet_replica.cc:330] stopping tablet replica
I20260929 16:28:46.236265 14891 raft_consensus.cc:2229] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:28:46.236560 14891 raft_consensus.cc:2258] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:46.237710 14891 ts_tablet_manager.cc:1905] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:28:46.241636 14891 ts_tablet_manager.cc:1918] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:28:46.241772 14891 log.cc:1196] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/wal/wals/96feb4f0b99b43a9b6c7ab153ed9cb83
I20260929 16:28:46.242069 14891 ts_tablet_manager.cc:1939] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 9f4683766c12485f873f8103679e5ff5: Deleting consensus metadata
I20260929 16:28:46.244199 14678 catalog_manager.cc:4841] TS 9f4683766c12485f873f8103679e5ff5 (127.0.98.1:35989): tablet 96feb4f0b99b43a9b6c7ab153ed9cb83 (table test-table [id=b8bdc250712e4d138d7be4d362d4f67a]) successfully deleted
W20260929 16:28:46.329095 14886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:46.329476 14886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:46.329524 14886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:46.333984 14886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:46.334053 14886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:46.334151 14886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:28:46.339797 14886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43893
--tserver_master_addrs=127.0.98.62:39305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:46.340982 14886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:46.342116 14886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:46.352221 14897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:46.352221 14896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:46.352234 14899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:46.352661 14886 server_base.cc:1034] running on GCE node
I20260929 16:28:46.353063 14886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:46.353801 14886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:46.355028 14886 hybrid_clock.cc:648] HybridClock initialized: now 1790699326355011 us; error 37 us; skew 500 ppm
Sep 29 16:28:46 dist-test-slave-vq9v krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:46.359067 14886 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:28:46.360384 14886 webserver.cc:460] Webserver started at http://127.0.98.2:43893/ using document root <none> and password file <none>
I20260929 16:28:46.361202 14886 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:46.361344 14886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:46.367172 14886 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:28:46.370066 14906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:46.371832 14886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:28:46.371999 14886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "738f451eb9654817824e91c33026f856"
format_stamp: "Formatted at 2026-09-29 16:28:45 on dist-test-slave-vq9v"
server_key: "c8bc4434c993c2186ee4952446bf1e4c"
server_key_iv: "5a7c00c78d26ac3d5d2f0fc5efb7d23a"
server_key_version: "encryptionkey@0"
I20260929 16:28:46.372834 14886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:46.423219 14886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:46.427337 14886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:46.427609 14886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:46.428293 14886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:28:46.429682 14913 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 16:28:46.434468 14886 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 16:28:46.434581 14886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260929 16:28:46.434674 14886 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 16:28:46.436620 14886 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 16:28:46.436703 14886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 16:28:46.437031 14913 tablet_bootstrap.cc:492] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856: Bootstrap starting.
I20260929 16:28:46.448091 14886 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40959
I20260929 16:28:46.448153 15020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40959 every 8 connection(s)
I20260929 16:28:46.449180 14886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:28:46.453974 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 14886
I20260929 16:28:46.454480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35689
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39305
--webserver_port=40839
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:28:46.458407 14913 log.cc:824] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:28:46 dist-test-slave-vq9v krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:28:46.466764 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.451540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53311 (local address 127.0.98.62:39305)
0929 16:28:46.451713 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:46.451718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:46.452465 (+ 747us) server_negotiation.cc:407] Connection header received
0929 16:28:46.453857 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:46.453861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:46.453987 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:46.454104 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:46.456060 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.456604 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.457950 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.458304 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.458442 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.462010 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:46.462015 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:46.462018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:46.462053 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:46.463970 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:46.464745 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:46.464748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:46.464750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:46.464849 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:46.465347 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:46.465350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:46.465352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:46.465628 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:46.465729 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:46.466456 (+ 727us) server_negotiation.cc:299] Negotiation successful
0929 16:28:46.466600 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:28:46.467449 14913 tablet_bootstrap.cc:492] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:28:46.467940 14913 tablet_bootstrap.cc:492] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856: Bootstrap complete.
I20260929 16:28:46.468163 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.451695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.2:53311)
0929 16:28:46.452247 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:46.452309 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:46.453574 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:46.454447 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:46.454458 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:46.455037 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:46.455813 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.455843 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.456766 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.456769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.457792 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.457803 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.458519 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.458526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.458785 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.459558 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:46.459587 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:46.461766 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:46.464122 (+ 2356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:46.464132 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:46.464150 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:46.464599 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:46.464962 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:46.464968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:46.464973 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:46.465239 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:46.465754 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:46.465764 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:46.466141 (+ 377us) client_negotiation.cc:769] Sending connection context
0929 16:28:46.466374 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 16:28:46.467025 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":146,"threads_started":1}
I20260929 16:28:46.469385 14913 ts_tablet_manager.cc:1397] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.007s
I20260929 16:28:46.469411 15021 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39305
I20260929 16:28:46.469631 15021 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:46.470192 15021 heartbeater.cc:511] Master 127.0.98.62:39305 requested a full tablet report, sending...
I20260929 16:28:46.472280 14693 ts_manager.cc:194] Registered new tserver with Master: 738f451eb9654817824e91c33026f856 (127.0.98.2:40959)
I20260929 16:28:46.472469 14913 raft_consensus.cc:348] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:46.473045 14913 raft_consensus.cc:729] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 738f451eb9654817824e91c33026f856, State: Initialized, Role: FOLLOWER
I20260929 16:28:46.473618 14913 consensus_queue.cc:260] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:46.473693 14693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53311
I20260929 16:28:46.474751 14913 ts_tablet_manager.cc:1428] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:28:46.476840 15021 heartbeater.cc:503] Master 127.0.98.62:39305 was elected leader, sending a full tablet report...
Sep 29 16:28:46 dist-test-slave-vq9v krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260929 16:28:46.484815 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.474457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51170 (local address 127.0.98.2:40959)
0929 16:28:46.474779 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:46.474790 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:46.474865 (+ 75us) server_negotiation.cc:407] Connection header received
0929 16:28:46.474936 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:46.474942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:46.475151 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:46.475287 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:46.475782 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.476992 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.478092 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.478587 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.478698 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.481077 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:46.481085 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:46.481097 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:46.481120 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:46.482922 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:46.483456 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:46.483461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:46.483463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:46.483548 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:46.483841 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:46.483843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:46.483845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:46.484140 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:46.484246 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:46.484554 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 16:28:46.484695 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":118,"threads_started":1}
I20260929 16:28:46.484813 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.474182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40959 (local address 127.0.0.1:51170)
0929 16:28:46.474331 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:46.474349 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:46.474447 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:46.475304 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:46.475307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:46.475541 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:46.475646 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.475654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.477118 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.477122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.477966 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.477976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.478704 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.478722 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.478828 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.479362 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:46.479376 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:46.480915 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:46.483046 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:46.483049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:46.483052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:46.483302 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:46.483648 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:46.483651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:46.483653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:46.483749 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:46.484247 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:46.484250 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:46.484361 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 16:28:46.484479 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 16:28:46.484637 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 16:28:46.486284 14950 tablet_service.cc:1514] Processing DeleteTablet for tablet 96feb4f0b99b43a9b6c7ab153ed9cb83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51170
I20260929 16:28:46.487272 15028 tablet_replica.cc:330] stopping tablet replica
I20260929 16:28:46.487581 15028 raft_consensus.cc:2229] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:28:46.487836 15028 raft_consensus.cc:2258] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:46.488818 15028 ts_tablet_manager.cc:1905] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:28:46.492506 15028 ts_tablet_manager.cc:1918] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:28:46.492630 15028 log.cc:1196] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/96feb4f0b99b43a9b6c7ab153ed9cb83
I20260929 16:28:46.492908 15028 ts_tablet_manager.cc:1939] T 96feb4f0b99b43a9b6c7ab153ed9cb83 P 738f451eb9654817824e91c33026f856: Deleting consensus metadata
I20260929 16:28:46.494580 14679 catalog_manager.cc:4841] TS 738f451eb9654817824e91c33026f856 (127.0.98.2:40959): tablet 96feb4f0b99b43a9b6c7ab153ed9cb83 (table test-table [id=b8bdc250712e4d138d7be4d362d4f67a]) successfully deleted
W20260929 16:28:46.578720 15024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:46.579013 15024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:46.579061 15024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:46.583334 15024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:46.583402 15024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:46.583464 15024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:28:46.588999 15024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40839
--tserver_master_addrs=127.0.98.62:39305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:46.590193 15024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:46.591259 15024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:46.601586 15033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:46.601656 15034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:46.601768 15024 server_base.cc:1034] running on GCE node
W20260929 16:28:46.601586 15036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:46.602452 15024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:46.603261 15024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:46.604506 15024 hybrid_clock.cc:648] HybridClock initialized: now 1790699326604485 us; error 40 us; skew 500 ppm
Sep 29 16:28:46 dist-test-slave-vq9v krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:46.608008 15024 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:28:46.609174 15024 webserver.cc:460] Webserver started at http://127.0.98.3:40839/ using document root <none> and password file <none>
I20260929 16:28:46.609925 15024 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:46.610059 15024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:46.614674 15024 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 16:28:46.617731 15043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:46.619205 15024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:28:46.619385 15024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "58e3238aaeea4f328a9eb827497df008"
format_stamp: "Formatted at 2026-09-29 16:28:45 on dist-test-slave-vq9v"
server_key: "84cb7047170beafa67f580be09ad4680"
server_key_iv: "0d0d570a89d060bb4816dbaae7434859"
server_key_version: "encryptionkey@0"
I20260929 16:28:46.620004 15024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:46.646871 15024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:46.650614 15024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:46.650839 15024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:46.651477 15024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:28:46.652346 15024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:46.652419 15024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:46.652490 15024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:46.652540 15024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:46.662461 15024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35689
I20260929 16:28:46.662508 15156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35689 every 8 connection(s)
I20260929 16:28:46.663537 15024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:28:46.673646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 15024
Sep 29 16:28:46 dist-test-slave-vq9v krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:28:46.679294 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.665576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56385 (local address 127.0.98.62:39305)
0929 16:28:46.665739 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:46.665743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:46.666682 (+ 939us) server_negotiation.cc:407] Connection header received
0929 16:28:46.668119 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:46.668123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:46.668189 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:46.668280 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:46.669785 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.670390 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.671705 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.672007 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.672114 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.675144 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:46.675148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:46.675151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:46.675179 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:46.676908 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:46.677572 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:46.677576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:46.677578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:46.677669 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:46.678070 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:46.678074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:46.678076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:46.678339 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:46.678445 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:46.678999 (+ 554us) server_negotiation.cc:299] Negotiation successful
0929 16:28:46.679129 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:28:46.680265 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.665973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.3:56385)
0929 16:28:46.666445 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:46.666537 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:46.667854 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:46.668529 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:46.668537 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:46.668971 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:46.669610 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.669626 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.670542 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.670545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.671553 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.671563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.672183 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.672186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.672415 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.673064 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:46.673082 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:46.674965 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:46.677061 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:46.677070 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:46.677092 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:46.677460 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:46.677767 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:46.677770 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:46.677772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:46.677950 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:46.678448 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:46.678457 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:46.678740 (+ 283us) client_negotiation.cc:769] Sending connection context
0929 16:28:46.678943 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 16:28:46.679297 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":116,"threads_started":1}
I20260929 16:28:46.681603 15157 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39305
I20260929 16:28:46.681892 15157 heartbeater.cc:464] Registering TS with master...
I20260929 16:28:46.682551 15157 heartbeater.cc:511] Master 127.0.98.62:39305 requested a full tablet report, sending...
I20260929 16:28:46.683734 14693 ts_manager.cc:194] Registered new tserver with Master: 58e3238aaeea4f328a9eb827497df008 (127.0.98.3:35689)
I20260929 16:28:46.683739 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.674546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59344 (local address 127.0.98.62:39305)
0929 16:28:46.674798 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:46.674802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:46.674963 (+ 161us) server_negotiation.cc:407] Connection header received
0929 16:28:46.675170 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:46.675178 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:46.675223 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:46.675292 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:46.676254 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.676791 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.677862 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.678120 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.678256 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.679785 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:46.679789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:46.679791 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:46.679823 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:46.681503 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:46.682035 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:46.682038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:46.682040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:46.682126 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:46.682430 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:46.682433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:46.682435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:46.682660 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:46.682853 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:46.683486 (+ 633us) server_negotiation.cc:299] Negotiation successful
0929 16:28:46.683619 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
I20260929 16:28:46.684362 14693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56385
I20260929 16:28:46.686275 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:28:46.691045 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.686909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59346 (local address 127.0.98.62:39305)
0929 16:28:46.687020 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:46.687024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:46.687225 (+ 201us) server_negotiation.cc:407] Connection header received
0929 16:28:46.687354 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:46.687356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:46.687508 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:46.687620 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:28:46.687965 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.688451 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.689455 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.689728 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.689823 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.690227 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:28:46.690657 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:28:46.690762 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:46.690868 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 16:28:46.690927 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 16:28:46.691766 14693 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:28:46.695515 14693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:28:46.704196 14816 tablet_service.cc:1467] Processing CreateTablet for tablet 951a02fa47a74548b356c0f9d692d788 (DEFAULT_TABLE table=test-table [id=6104e77fe7604bbb85c052cc3dba30f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:46.704213 14950 tablet_service.cc:1467] Processing CreateTablet for tablet 951a02fa47a74548b356c0f9d692d788 (DEFAULT_TABLE table=test-table [id=6104e77fe7604bbb85c052cc3dba30f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:46.704860 14816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 951a02fa47a74548b356c0f9d692d788. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:46.704860 14950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 951a02fa47a74548b356c0f9d692d788. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:46.707849 14774 tablet_bootstrap.cc:492] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5: Bootstrap starting.
I20260929 16:28:46.707918 14913 tablet_bootstrap.cc:492] T 951a02fa47a74548b356c0f9d692d788 P 738f451eb9654817824e91c33026f856: Bootstrap starting.
I20260929 16:28:46.708866 14774 tablet_bootstrap.cc:654] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:46.708877 14913 tablet_bootstrap.cc:654] T 951a02fa47a74548b356c0f9d692d788 P 738f451eb9654817824e91c33026f856: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:46.708916 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.703443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35689 (local address 127.0.0.1:51886)
0929 16:28:46.703711 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:46.703728 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:46.703805 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:46.704544 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:46.704547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:46.704568 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:46.704678 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.704683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.706091 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.706097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.707160 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:46.707171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.708366 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.708370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:46.708491 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.708551 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 16:28:46.708636 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 16:28:46.708732 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260929 16:28:46.710214 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:46.703649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51886 (local address 127.0.98.3:35689)
0929 16:28:46.704041 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:46.704055 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:46.704082 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:28:46.704159 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:46.704163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:46.704342 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:46.704816 (+ 474us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:46.704990 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.705936 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.707298 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:46.708179 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:46.708332 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:46.708496 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:46.709851 (+ 1355us) server_negotiation.cc:299] Negotiation successful
0929 16:28:46.710007 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":232,"threads_started":1}
I20260929 16:28:46.710546 14913 tablet_bootstrap.cc:492] T 951a02fa47a74548b356c0f9d692d788 P 738f451eb9654817824e91c33026f856: No bootstrap required, opened a new log
I20260929 16:28:46.710563 14774 tablet_bootstrap.cc:492] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5: No bootstrap required, opened a new log
I20260929 16:28:46.710647 14913 ts_tablet_manager.cc:1397] T 951a02fa47a74548b356c0f9d692d788 P 738f451eb9654817824e91c33026f856: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 16:28:46.710672 14774 ts_tablet_manager.cc:1397] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 16:28:46.711103 14913 raft_consensus.cc:348] T 951a02fa47a74548b356c0f9d692d788 P 738f451eb9654817824e91c33026f856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:46.711083 14774 raft_consensus.cc:348] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:46.711230 14774 raft_consensus.cc:374] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:46.711229 14913 raft_consensus.cc:374] T 951a02fa47a74548b356c0f9d692d788 P 738f451eb9654817824e91c33026f856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:46.711274 14774 raft_consensus.cc:729] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f4683766c12485f873f8103679e5ff5, State: Initialized, Role: FOLLOWER
I20260929 16:28:46.711280 14913 raft_consensus.cc:729] T 951a02fa47a74548b356c0f9d692d788 P 738f451eb9654817824e91c33026f856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 738f451eb9654817824e91c33026f856, State: Initialized, Role: FOLLOWER
I20260929 16:28:46.711397 14774 consensus_queue.cc:260] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:46.711426 14913 consensus_queue.cc:260] T 951a02fa47a74548b356c0f9d692d788 P 738f451eb9654817824e91c33026f856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:46.711658 14774 ts_tablet_manager.cc:1428] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:28:46.711704 14913 ts_tablet_manager.cc:1428] T 951a02fa47a74548b356c0f9d692d788 P 738f451eb9654817824e91c33026f856: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 16:28:46.714030 15091 tablet_service.cc:1467] Processing CreateTablet for tablet 951a02fa47a74548b356c0f9d692d788 (DEFAULT_TABLE table=test-table [id=6104e77fe7604bbb85c052cc3dba30f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:28:46.715478 15091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 951a02fa47a74548b356c0f9d692d788. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:28:46.723918 15169 tablet_bootstrap.cc:492] T 951a02fa47a74548b356c0f9d692d788 P 58e3238aaeea4f328a9eb827497df008: Bootstrap starting.
I20260929 16:28:46.726370 15169 tablet_bootstrap.cc:654] T 951a02fa47a74548b356c0f9d692d788 P 58e3238aaeea4f328a9eb827497df008: Neither blocks nor log segments found. Creating new log.
I20260929 16:28:46.727633 15169 log.cc:824] T 951a02fa47a74548b356c0f9d692d788 P 58e3238aaeea4f328a9eb827497df008: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:46.730790 15169 tablet_bootstrap.cc:492] T 951a02fa47a74548b356c0f9d692d788 P 58e3238aaeea4f328a9eb827497df008: No bootstrap required, opened a new log
I20260929 16:28:46.731048 15169 ts_tablet_manager.cc:1397] T 951a02fa47a74548b356c0f9d692d788 P 58e3238aaeea4f328a9eb827497df008: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260929 16:28:46.734426 15169 raft_consensus.cc:348] T 951a02fa47a74548b356c0f9d692d788 P 58e3238aaeea4f328a9eb827497df008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:46.734714 15169 raft_consensus.cc:374] T 951a02fa47a74548b356c0f9d692d788 P 58e3238aaeea4f328a9eb827497df008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:28:46.734748 15169 raft_consensus.cc:729] T 951a02fa47a74548b356c0f9d692d788 P 58e3238aaeea4f328a9eb827497df008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e3238aaeea4f328a9eb827497df008, State: Initialized, Role: FOLLOWER
I20260929 16:28:46.735270 15169 consensus_queue.cc:260] T 951a02fa47a74548b356c0f9d692d788 P 58e3238aaeea4f328a9eb827497df008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:46.736059 15157 heartbeater.cc:503] Master 127.0.98.62:39305 was elected leader, sending a full tablet report...
I20260929 16:28:46.736364 15169 ts_tablet_manager.cc:1428] T 951a02fa47a74548b356c0f9d692d788 P 58e3238aaeea4f328a9eb827497df008: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260929 16:28:46.914651 15158 tablet.cc:2369] T 951a02fa47a74548b356c0f9d692d788 P 58e3238aaeea4f328a9eb827497df008: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:28:46.937824 14883 tablet.cc:2369] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:28:46.950747 15022 tablet.cc:2369] T 951a02fa47a74548b356c0f9d692d788 P 738f451eb9654817824e91c33026f856: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:28:47.016494 14888 raft_consensus.cc:482] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:28:47.016810 14888 raft_consensus.cc:504] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:47.018164 14888 leader_election.cc:288] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58e3238aaeea4f328a9eb827497df008 (127.0.98.3:35689), 738f451eb9654817824e91c33026f856 (127.0.98.2:40959)
I20260929 16:28:47.024330 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.018566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52487 (local address 127.0.98.2:40959)
0929 16:28:47.018871 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:47.018874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:47.018983 (+ 109us) server_negotiation.cc:407] Connection header received
0929 16:28:47.019285 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:47.019288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:47.019356 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:47.019490 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:47.019888 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.020701 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.022761 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.023480 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.023598 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.023816 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:47.024103 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 16:28:47.024195 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":77,"threads_started":1}
I20260929 16:28:47.024329 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.018602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50707 (local address 127.0.98.3:35689)
0929 16:28:47.018745 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:47.018750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:47.019054 (+ 304us) server_negotiation.cc:407] Connection header received
0929 16:28:47.019285 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:47.019288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:47.019341 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:47.019442 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:28:47.019888 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.020688 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.022761 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.023465 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.023598 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.023667 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:47.024103 (+ 436us) server_negotiation.cc:299] Negotiation successful
0929 16:28:47.024190 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:28:47.024364 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.018475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35689 (local address 127.0.98.1:50707)
0929 16:28:47.018968 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:47.018981 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:47.019053 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:47.019533 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:47.019537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:47.019584 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:47.019773 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:47.019778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.020825 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.020829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:47.022597 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:47.022607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.023642 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.023647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:47.023938 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.023956 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 16:28:47.024052 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 16:28:47.024172 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":358,"spinlock_wait_cycles":12160,"thread_start_us":82,"threads_started":1}
I20260929 16:28:47.024366 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.018473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40959 (local address 127.0.98.1:52487)
0929 16:28:47.018869 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:47.018885 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:47.019033 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:47.019533 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:47.019537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:47.019583 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:28:47.019760 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:47.019769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.020825 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.020829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:47.022594 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:47.022607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.023642 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.023647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:47.023898 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.023957 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 16:28:47.024102 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 16:28:47.024186 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":411,"spinlock_wait_cycles":32000,"thread_start_us":88,"threads_started":1}
I20260929 16:28:47.025195 15111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951a02fa47a74548b356c0f9d692d788" candidate_uuid: "9f4683766c12485f873f8103679e5ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e3238aaeea4f328a9eb827497df008" is_pre_election: true
I20260929 16:28:47.025544 14975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951a02fa47a74548b356c0f9d692d788" candidate_uuid: "9f4683766c12485f873f8103679e5ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "738f451eb9654817824e91c33026f856" is_pre_election: true
I20260929 16:28:47.025633 15111 raft_consensus.cc:2454] T 951a02fa47a74548b356c0f9d692d788 P 58e3238aaeea4f328a9eb827497df008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f4683766c12485f873f8103679e5ff5 in term 0.
I20260929 16:28:47.025867 14975 raft_consensus.cc:2454] T 951a02fa47a74548b356c0f9d692d788 P 738f451eb9654817824e91c33026f856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f4683766c12485f873f8103679e5ff5 in term 0.
I20260929 16:28:47.026324 14769 leader_election.cc:302] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 58e3238aaeea4f328a9eb827497df008, 738f451eb9654817824e91c33026f856, 9f4683766c12485f873f8103679e5ff5; no voters:
I20260929 16:28:47.026690 14888 raft_consensus.cc:2781] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:28:47.026798 14888 raft_consensus.cc:482] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:28:47.026876 14888 raft_consensus.cc:3037] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:47.027966 14888 raft_consensus.cc:504] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:47.028355 14888 leader_election.cc:288] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 58e3238aaeea4f328a9eb827497df008 (127.0.98.3:35689), 738f451eb9654817824e91c33026f856 (127.0.98.2:40959)
I20260929 16:28:47.028846 14975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951a02fa47a74548b356c0f9d692d788" candidate_uuid: "9f4683766c12485f873f8103679e5ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "738f451eb9654817824e91c33026f856"
I20260929 16:28:47.028967 15111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951a02fa47a74548b356c0f9d692d788" candidate_uuid: "9f4683766c12485f873f8103679e5ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e3238aaeea4f328a9eb827497df008"
I20260929 16:28:47.029047 14975 raft_consensus.cc:3037] T 951a02fa47a74548b356c0f9d692d788 P 738f451eb9654817824e91c33026f856 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:47.029150 15111 raft_consensus.cc:3037] T 951a02fa47a74548b356c0f9d692d788 P 58e3238aaeea4f328a9eb827497df008 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:28:47.030122 14975 raft_consensus.cc:2454] T 951a02fa47a74548b356c0f9d692d788 P 738f451eb9654817824e91c33026f856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f4683766c12485f873f8103679e5ff5 in term 1.
I20260929 16:28:47.030187 15111 raft_consensus.cc:2454] T 951a02fa47a74548b356c0f9d692d788 P 58e3238aaeea4f328a9eb827497df008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f4683766c12485f873f8103679e5ff5 in term 1.
I20260929 16:28:47.030488 14769 leader_election.cc:302] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 738f451eb9654817824e91c33026f856, 9f4683766c12485f873f8103679e5ff5; no voters:
I20260929 16:28:47.030694 14888 raft_consensus.cc:2781] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:28:47.030932 14888 raft_consensus.cc:686] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 9f4683766c12485f873f8103679e5ff5, State: Running, Role: LEADER
I20260929 16:28:47.031204 14888 consensus_queue.cc:237] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:47.034395 14694 catalog_manager.cc:5495] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f4683766c12485f873f8103679e5ff5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9f4683766c12485f873f8103679e5ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } health_report { overall_health: UNKNOWN } } }
I20260929 16:28:47.080236 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.076232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45046 (local address 127.0.98.1:35989)
0929 16:28:47.076518 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:47.076526 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:47.076539 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:28:47.076614 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:47.076616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:47.076667 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:47.076765 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:28:47.077093 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.077679 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.078723 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:47.078952 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:47.079040 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:47.079315 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:28:47.079764 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:28:47.079851 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:47.080046 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 16:28:47.080098 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":120,"threads_started":1}
I20260929 16:28:47.084466 14975 raft_consensus.cc:1261] T 951a02fa47a74548b356c0f9d692d788 P 738f451eb9654817824e91c33026f856 [term 1 FOLLOWER]: Refusing update from remote peer 9f4683766c12485f873f8103679e5ff5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:47.084470 15111 raft_consensus.cc:1261] T 951a02fa47a74548b356c0f9d692d788 P 58e3238aaeea4f328a9eb827497df008 [term 1 FOLLOWER]: Refusing update from remote peer 9f4683766c12485f873f8103679e5ff5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:28:47.085357 14888 consensus_queue.cc:1035] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:28:47.085635 15175 consensus_queue.cc:1035] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:28:47.090694 15178 mvcc.cc:205] Tried to move back new op lower bound from 7334704443731083264 to 7334704443522355200. Current Snapshot: MvccSnapshot[applied={T|T < 7334704443731083264}]
I20260929 16:28:47.092247 15179 mvcc.cc:205] Tried to move back new op lower bound from 7334704443731083264 to 7334704443522355200. Current Snapshot: MvccSnapshot[applied={T|T < 7334704443731083264}]
I20260929 16:28:47.100382 14693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:28:47.100577 14693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:28:47.102941 14693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3: Sending DeleteTablet for 3 replicas of tablet 951a02fa47a74548b356c0f9d692d788
I20260929 16:28:47.103717 14816 tablet_service.cc:1514] Processing DeleteTablet for tablet 951a02fa47a74548b356c0f9d692d788 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45034
I20260929 16:28:47.103739 15091 tablet_service.cc:1514] Processing DeleteTablet for tablet 951a02fa47a74548b356c0f9d692d788 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51886
I20260929 16:28:47.104039 14950 tablet_service.cc:1514] Processing DeleteTablet for tablet 951a02fa47a74548b356c0f9d692d788 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:28:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51170
I20260929 16:28:47.104269 15189 tablet_replica.cc:330] T 951a02fa47a74548b356c0f9d692d788 P 58e3238aaeea4f328a9eb827497df008: stopping tablet replica
I20260929 16:28:47.104475 15190 tablet_replica.cc:330] T 951a02fa47a74548b356c0f9d692d788 P 738f451eb9654817824e91c33026f856: stopping tablet replica
I20260929 16:28:47.104630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 14732
I20260929 16:28:47.104627 15190 raft_consensus.cc:2229] T 951a02fa47a74548b356c0f9d692d788 P 738f451eb9654817824e91c33026f856 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:28:47.104645 15189 raft_consensus.cc:2229] T 951a02fa47a74548b356c0f9d692d788 P 58e3238aaeea4f328a9eb827497df008 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:28:47.104868 15190 raft_consensus.cc:2258] T 951a02fa47a74548b356c0f9d692d788 P 738f451eb9654817824e91c33026f856 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:47.105543 15189 raft_consensus.cc:2258] T 951a02fa47a74548b356c0f9d692d788 P 58e3238aaeea4f328a9eb827497df008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:28:47.106659 15190 ts_tablet_manager.cc:1905] T 951a02fa47a74548b356c0f9d692d788 P 738f451eb9654817824e91c33026f856: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:28:47.107237 15189 ts_tablet_manager.cc:1905] T 951a02fa47a74548b356c0f9d692d788 P 58e3238aaeea4f328a9eb827497df008: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:28:47.109483 15190 ts_tablet_manager.cc:1918] T 951a02fa47a74548b356c0f9d692d788 P 738f451eb9654817824e91c33026f856: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:28:47.109576 15190 log.cc:1196] T 951a02fa47a74548b356c0f9d692d788 P 738f451eb9654817824e91c33026f856: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/951a02fa47a74548b356c0f9d692d788
I20260929 16:28:47.109833 15190 ts_tablet_manager.cc:1939] T 951a02fa47a74548b356c0f9d692d788 P 738f451eb9654817824e91c33026f856: Deleting consensus metadata
I20260929 16:28:47.110260 15189 ts_tablet_manager.cc:1918] T 951a02fa47a74548b356c0f9d692d788 P 58e3238aaeea4f328a9eb827497df008: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:28:47.110416 15189 log.cc:1196] T 951a02fa47a74548b356c0f9d692d788 P 58e3238aaeea4f328a9eb827497df008: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/951a02fa47a74548b356c0f9d692d788
I20260929 16:28:47.110770 15189 ts_tablet_manager.cc:1939] T 951a02fa47a74548b356c0f9d692d788 P 58e3238aaeea4f328a9eb827497df008: Deleting consensus metadata
I20260929 16:28:47.110756 14679 catalog_manager.cc:4841] TS 738f451eb9654817824e91c33026f856 (127.0.98.2:40959): tablet 951a02fa47a74548b356c0f9d692d788 (table test-table [id=6104e77fe7604bbb85c052cc3dba30f5]) successfully deleted
W20260929 16:28:47.111721 14678 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35989 (error 108)
I20260929 16:28:47.111982 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 14886
I20260929 16:28:47.112594 14681 catalog_manager.cc:4841] TS 58e3238aaeea4f328a9eb827497df008 (127.0.98.3:35689): tablet 951a02fa47a74548b356c0f9d692d788 (table test-table [id=6104e77fe7604bbb85c052cc3dba30f5]) successfully deleted
I20260929 16:28:47.113385 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.112982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35989 (local address 127.0.0.1:45048)
0929 16:28:47.113152 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:47.113269 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 16:28:47.113590 14678 catalog_manager.cc:4584] TS 9f4683766c12485f873f8103679e5ff5 (127.0.98.1:35989): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 951a02fa47a74548b356c0f9d692d788: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35989: connect: Connection refused (error 111)
I20260929 16:28:47.118675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 15024
I20260929 16:28:47.124465 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 14661
I20260929 16:28:47.131085 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39305
--webserver_interface=127.0.98.62
--webserver_port=41621
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:28:47.246544 15192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:47.246842 15192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:47.246887 15192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:47.251165 15192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:28:47.251248 15192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:47.251268 15192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:47.251286 15192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:28:47.251302 15192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:28:47.257091 15192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:47.258334 15192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:47.259449 15192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:47.269047 15198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:47.269266 15192 server_base.cc:1034] running on GCE node
W20260929 16:28:47.269060 15200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:47.269060 15197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:47.269974 15192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:47.271203 15192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:47.272397 15192 hybrid_clock.cc:648] HybridClock initialized: now 1790699327272384 us; error 31 us; skew 500 ppm
Sep 29 16:28:47 dist-test-slave-vq9v krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:47.275594 15192 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:28:47.276654 15192 webserver.cc:460] Webserver started at http://127.0.98.62:41621/ using document root <none> and password file <none>
I20260929 16:28:47.277384 15192 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:47.277488 15192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:47.281811 15192 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:28:47.284411 15207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:47.286053 15192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:28:47.286229 15192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3"
format_stamp: "Formatted at 2026-09-29 16:28:44 on dist-test-slave-vq9v"
server_key: "735567f8adcd83831254c7e8e7e09893"
server_key_iv: "e63080736585eb95bcc662142d2e30eb"
server_key_version: "encryptionkey@0"
I20260929 16:28:47.286948 15192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:47.312511 15192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:47.316248 15192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:47.316483 15192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:47.324532 15192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39305
I20260929 16:28:47.324555 15259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39305 every 8 connection(s)
I20260929 16:28:47.325729 15192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:28:47.328261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 15192
I20260929 16:28:47.328768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35989
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39305
--webserver_port=37177
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:28:47.330456 15260 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:28:47.332520 15260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3: Bootstrap starting.
I20260929 16:28:47.356849 15260 log.cc:824] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:47.369552 15260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:28:47.370003 15260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3: Bootstrap complete.
I20260929 16:28:47.374769 15260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } }
I20260929 16:28:47.375260 15260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b9ba30cc08a4dad9ea3b0b818094dc3, State: Initialized, Role: FOLLOWER
I20260929 16:28:47.375815 15260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } }
I20260929 16:28:47.375954 15260 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:28:47.376049 15260 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:28:47.376170 15260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 2 FOLLOWER]: Advancing to term 3
I20260929 16:28:47.378432 15260 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } }
I20260929 16:28:47.378818 15260 leader_election.cc:302] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b9ba30cc08a4dad9ea3b0b818094dc3; no voters:
I20260929 16:28:47.379345 15260 leader_election.cc:288] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 16:28:47.379462 15264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 16:28:47.379673 15264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [term 3 LEADER]: Becoming Leader. State: Replica: 0b9ba30cc08a4dad9ea3b0b818094dc3, State: Running, Role: LEADER
I20260929 16:28:47.380007 15264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } }
I20260929 16:28:47.380697 15260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:28:47.381469 15265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } } }
I20260929 16:28:47.381616 15265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:47.382244 15266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b9ba30cc08a4dad9ea3b0b818094dc3. Latest consensus state: current_term: 3 leader_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9ba30cc08a4dad9ea3b0b818094dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39305 } } }
I20260929 16:28:47.382381 15266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9ba30cc08a4dad9ea3b0b818094dc3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:28:47.382772 15273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:28:47.388129 15273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:28:47.388839 15273 catalog_manager.cc:1259] Loaded cluster ID: 912a670d935a45d1af400c0f82fc1ef2
I20260929 16:28:47.388906 15273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 16:28:47.389957 15273 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f321dbfc9ad google::LogMessage::Fail() at ??:0
@ 0x7f321dbff88a google::LogMessage::SendToLog() at ??:0
@ 0x7f321dbfc4de google::LogMessage::Flush() at ??:0
@ 0x7f321dbfda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f322853bfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f322853cd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3228539579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f322857d640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f321fe4e3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f321fe4ec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f321fe503f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f321fe3f512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f32214836db start_thread at ??:0
@ 0x7f321c6fa61f clone at ??:0
W20260929 16:28:47.492110 15262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:47.492409 15262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:47.492455 15262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:47.496980 15262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:47.497061 15262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:47.497126 15262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:28:47.503062 15262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37177
--tserver_master_addrs=127.0.98.62:39305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:47.504233 15262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:47.505393 15262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:47.516352 15301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:47.516506 15302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:47.516345 15304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:47.517378 15262 server_base.cc:1034] running on GCE node
I20260929 16:28:47.517952 15262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:47.518774 15262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:47.519982 15262 hybrid_clock.cc:648] HybridClock initialized: now 1790699327519952 us; error 67 us; skew 500 ppm
Sep 29 16:28:47 dist-test-slave-vq9v krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:47.523396 15262 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:28:47.524520 15262 webserver.cc:460] Webserver started at http://127.0.98.1:37177/ using document root <none> and password file <none>
I20260929 16:28:47.525161 15262 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:47.525314 15262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:47.532060 15262 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:28:47.534680 15311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:47.536202 15262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:28:47.536386 15262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "9f4683766c12485f873f8103679e5ff5"
format_stamp: "Formatted at 2026-09-29 16:28:45 on dist-test-slave-vq9v"
server_key: "ef58146c263e36f2c8f7accb76fd5051"
server_key_iv: "d77357fd425afd115983d6f908879908"
server_key_version: "encryptionkey@0"
I20260929 16:28:47.537065 15262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:47.556674 15262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:47.560464 15262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:47.560704 15262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:47.561527 15262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:28:47.563141 15318 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 16:28:47.569008 15262 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 16:28:47.569133 15262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 16:28:47.569252 15262 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 16:28:47.571151 15262 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 16:28:47.571239 15262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 16:28:47.571692 15318 tablet_bootstrap.cc:492] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5: Bootstrap starting.
I20260929 16:28:47.582608 15262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35989
I20260929 16:28:47.583017 15425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35989 every 8 connection(s)
I20260929 16:28:47.583849 15262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:28:47.587461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 15262
I20260929 16:28:47.587900 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40959
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39305
--webserver_port=43893
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:28:47.593384 15318 log.cc:824] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5: Log is configured to *not* fsync() on all Append() calls
I20260929 16:28:47.604439 15318 tablet_bootstrap.cc:492] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:28:47.604928 15318 tablet_bootstrap.cc:492] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5: Bootstrap complete.
I20260929 16:28:47.606436 15318 ts_tablet_manager.cc:1397] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5: Time spent bootstrapping tablet: real 0.035s user 0.022s sys 0.012s
I20260929 16:28:47.610320 15318 raft_consensus.cc:348] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:47.611039 15318 raft_consensus.cc:729] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f4683766c12485f873f8103679e5ff5, State: Initialized, Role: FOLLOWER
I20260929 16:28:47.611783 15318 consensus_queue.cc:260] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:47.612963 15318 ts_tablet_manager.cc:1428] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
W20260929 16:28:47.638607 15428 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:28:47.586445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.1:34055)
0929 16:28:47.587116 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:47.587178 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:47.588468 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:47.637810 (+ 49342us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":176,"threads_started":1}
W20260929 16:28:47.639164 15426 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39305 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39305: Client connection negotiation failed: client connection to 127.0.98.62:39305: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 16:28:47.640034 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.639719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.1:41785)
0929 16:28:47.639875 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:47.639951 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 16:28:47.640810 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.640579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.1:59519)
0929 16:28:47.640694 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:47.640753 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 16:28:47.640983 15426 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:28:47.715415 15429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:47.715768 15429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:47.715859 15429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:47.720265 15429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:47.720393 15429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:47.720491 15429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:28:47.726658 15429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43893
--tserver_master_addrs=127.0.98.62:39305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:47.727841 15429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:47.728902 15429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:47.739146 15437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:47.739210 15439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:47.739146 15436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:47.739493 15429 server_base.cc:1034] running on GCE node
I20260929 16:28:47.740087 15429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:47.740936 15429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:47.742130 15429 hybrid_clock.cc:648] HybridClock initialized: now 1790699327742105 us; error 83 us; skew 500 ppm
Sep 29 16:28:47 dist-test-slave-vq9v krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:47.745167 15429 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:28:47.746281 15429 webserver.cc:460] Webserver started at http://127.0.98.2:43893/ using document root <none> and password file <none>
I20260929 16:28:47.746902 15429 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:47.747013 15429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:47.751394 15429 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 16:28:47.754086 15446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:47.755551 15429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:28:47.755743 15429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "738f451eb9654817824e91c33026f856"
format_stamp: "Formatted at 2026-09-29 16:28:45 on dist-test-slave-vq9v"
server_key: "c8bc4434c993c2186ee4952446bf1e4c"
server_key_iv: "5a7c00c78d26ac3d5d2f0fc5efb7d23a"
server_key_version: "encryptionkey@0"
I20260929 16:28:47.756413 15429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:47.770788 15429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:47.774411 15429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:47.774652 15429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:47.775300 15429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:28:47.776180 15429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:47.776257 15429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:47.776330 15429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:47.776376 15429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:47.785717 15429 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40959
I20260929 16:28:47.785732 15559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40959 every 8 connection(s)
I20260929 16:28:47.786774 15429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:28:47.788683 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 15429
I20260929 16:28:47.789237 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35689
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39305
--webserver_port=40839
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:28:47.791093 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.789261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.2:57449)
0929 16:28:47.789783 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:47.789893 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":147,"threads_started":1}
W20260929 16:28:47.791713 15560 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39305 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39305: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
I20260929 16:28:47.792413 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.792198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.2:39641)
0929 16:28:47.792312 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:47.792359 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 16:28:47.793123 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:47.792849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.2:37687)
0929 16:28:47.792982 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:47.793044 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:28:47.793402 15560 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:28:47.907771 15563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:28:47.908080 15563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:28:47.908125 15563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:28:47.912436 15563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:28:47.912511 15563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:28:47.912575 15563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:28:47.918182 15563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40839
--tserver_master_addrs=127.0.98.62:39305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:28:47.919325 15563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:28:47.920369 15563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:28:47.930871 15571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:47.930872 15568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:28:47.930966 15569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:28:47.931110 15563 server_base.cc:1034] running on GCE node
I20260929 16:28:47.931716 15563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:28:47.932511 15563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:28:47.933700 15563 hybrid_clock.cc:648] HybridClock initialized: now 1790699327933685 us; error 31 us; skew 500 ppm
Sep 29 16:28:47 dist-test-slave-vq9v krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:28:47.936772 15563 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:28:47.937922 15563 webserver.cc:460] Webserver started at http://127.0.98.3:40839/ using document root <none> and password file <none>
I20260929 16:28:47.938527 15563 fs_manager.cc:359] Metadata directory not provided
I20260929 16:28:47.938627 15563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:28:47.942986 15563 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:28:47.945530 15578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:47.947268 15563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:28:47.947453 15563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "58e3238aaeea4f328a9eb827497df008"
format_stamp: "Formatted at 2026-09-29 16:28:45 on dist-test-slave-vq9v"
server_key: "84cb7047170beafa67f580be09ad4680"
server_key_iv: "0d0d570a89d060bb4816dbaae7434859"
server_key_version: "encryptionkey@0"
I20260929 16:28:47.948117 15563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:28:47.993875 15563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:28:47.997426 15563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:28:47.997661 15563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:28:47.998286 15563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:28:47.999159 15563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:28:47.999236 15563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:47.999325 15563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:28:47.999372 15563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:28:48.009248 15563 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35689
I20260929 16:28:48.009323 15691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35689 every 8 connection(s)
I20260929 16:28:48.010355 15563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:28:48.014607 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:48.012732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.3:42743)
0929 16:28:48.013244 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:48.013371 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":125,"threads_started":1}
W20260929 16:28:48.015089 15692 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39305 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39305: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
I20260929 16:28:48.015692 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:48.015475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.3:38709)
0929 16:28:48.015588 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:48.015635 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 16:28:48.016440 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:48.016151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.3:43257)
0929 16:28:48.016344 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:48.016390 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260929 16:28:48.016723 15692 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:28:48.017680 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 15563
I20260929 16:28:48.642304 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:48.641777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.1:37505)
0929 16:28:48.642095 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:48.642161 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":75,"threads_started":1}
I20260929 16:28:48.794505 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:48.794045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.2:45867)
0929 16:28:48.794306 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:48.794368 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20260929 16:28:49.017931 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:49.017492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.3:43615)
0929 16:28:49.017758 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:49.017817 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":73,"threads_started":1}
I20260929 16:28:49.078797 15699 raft_consensus.cc:482] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:28:49.079095 15699 raft_consensus.cc:504] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:49.080282 15699 leader_election.cc:288] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58e3238aaeea4f328a9eb827497df008 (127.0.98.3:35689), 738f451eb9654817824e91c33026f856 (127.0.98.2:40959)
Sep 29 16:28:49 dist-test-slave-vq9v krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 29 16:28:49 dist-test-slave-vq9v krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260929 16:28:49.095497 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:49.080761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43981 (local address 127.0.98.3:35689)
0929 16:28:49.081337 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:49.081357 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:49.081391 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:28:49.082071 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:49.082077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:49.082412 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:49.082737 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:49.084139 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:49.085267 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:49.086529 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:49.087049 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:49.087240 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:49.090518 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:49.090523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:49.090553 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:49.090585 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:49.092586 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:49.093356 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:49.093361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:49.093365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:49.093478 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:49.093911 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:49.093914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:49.093915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:49.094321 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:49.094454 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:49.095034 (+ 580us) server_negotiation.cc:299] Negotiation successful
0929 16:28:49.095304 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":175,"threads_started":1}
I20260929 16:28:49.095485 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:49.080580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40959 (local address 127.0.98.1:45307)
0929 16:28:49.080775 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:49.080794 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:49.080932 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:49.082819 (+ 1887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:49.082826 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:49.083383 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:49.083920 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:49.083938 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:49.085329 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:49.085336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:49.086391 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:49.086402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:49.087180 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:49.087184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:49.087424 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:49.088131 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:49.088157 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:49.090634 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:49.092935 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:49.092937 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:49.092939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:49.093189 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:49.093641 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:49.093643 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:49.093644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:49.093757 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:49.094463 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:49.094469 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:49.094707 (+ 238us) client_negotiation.cc:769] Sending connection context
0929 16:28:49.094908 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 16:28:49.095291 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":243,"spinlock_wait_cycles":11904}
I20260929 16:28:49.095562 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:49.080737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45307 (local address 127.0.98.2:40959)
0929 16:28:49.081272 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0929 16:28:49.081304 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0929 16:28:49.081349 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:28:49.082040 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:28:49.082046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:28:49.082412 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:28:49.082707 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:28:49.084169 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:49.085140 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:49.086571 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:28:49.087040 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:28:49.087214 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:49.090855 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:28:49.090861 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:28:49.090873 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:28:49.090906 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:28:49.092791 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:49.093356 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:49.093362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:49.093370 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:49.093497 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:28:49.093905 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:28:49.093909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:28:49.093913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:28:49.094323 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:28:49.094484 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 16:28:49.095032 (+ 548us) server_negotiation.cc:299] Negotiation successful
0929 16:28:49.095342 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":116,"threads_started":1}
I20260929 16:28:49.095945 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:49.080580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35689 (local address 127.0.98.1:43981)
0929 16:28:49.081050 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:49.081066 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:28:49.081181 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:28:49.083106 (+ 1925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:28:49.083108 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:28:49.083367 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:28:49.083921 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:49.083937 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:49.085405 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:49.085407 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:49.086388 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:28:49.086398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:28:49.087202 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:28:49.087205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:28:49.087416 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:28:49.088183 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:28:49.088195 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:28:49.090233 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:28:49.092732 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:49.092739 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:49.092754 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:49.093186 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:49.093592 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:28:49.093595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:28:49.093597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:28:49.093771 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:28:49.094447 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:28:49.094454 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:28:49.094707 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 16:28:49.094909 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 16:28:49.095393 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":481,"spinlock_wait_cycles":29824,"thread_start_us":73,"threads_started":1}
W20260929 16:28:49.098194 15313 leader_election.cc:341] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 738f451eb9654817824e91c33026f856 (127.0.98.2:40959): Not found: Tablet not found: 951a02fa47a74548b356c0f9d692d788
W20260929 16:28:49.098582 15315 leader_election.cc:341] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58e3238aaeea4f328a9eb827497df008 (127.0.98.3:35689): Not found: Tablet not found: 951a02fa47a74548b356c0f9d692d788
I20260929 16:28:49.098671 15315 leader_election.cc:302] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f4683766c12485f873f8103679e5ff5; no voters: 58e3238aaeea4f328a9eb827497df008, 738f451eb9654817824e91c33026f856
I20260929 16:28:49.098944 15699 raft_consensus.cc:2726] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:28:49.643635 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:49.643189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.1:59435)
0929 16:28:49.643471 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:49.643536 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
I20260929 16:28:49.795821 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:49.795333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.2:34853)
0929 16:28:49.795656 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:49.795719 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":72,"threads_started":1}
I20260929 16:28:50.019418 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:50.018896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.3:43805)
0929 16:28:50.019216 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:50.019281 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":86,"threads_started":1}
I20260929 16:28:50.645269 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:50.644684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.1:41189)
0929 16:28:50.645009 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:50.645073 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":91,"threads_started":1}
I20260929 16:28:50.797143 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:50.796647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.2:46767)
0929 16:28:50.796964 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:50.797032 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":82,"threads_started":1}
I20260929 16:28:51.020659 15709 raft_consensus.cc:482] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:28:51.020817 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.020314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.3:56115)
0929 16:28:51.020611 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:51.020673 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":125,"threads_started":1}
I20260929 16:28:51.020782 15709 raft_consensus.cc:504] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:51.021304 15709 leader_election.cc:288] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58e3238aaeea4f328a9eb827497df008 (127.0.98.3:35689), 738f451eb9654817824e91c33026f856 (127.0.98.2:40959)
W20260929 16:28:51.022068 15315 leader_election.cc:341] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58e3238aaeea4f328a9eb827497df008 (127.0.98.3:35689): Not found: Tablet not found: 951a02fa47a74548b356c0f9d692d788
W20260929 16:28:51.022161 15313 leader_election.cc:341] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 738f451eb9654817824e91c33026f856 (127.0.98.2:40959): Not found: Tablet not found: 951a02fa47a74548b356c0f9d692d788
I20260929 16:28:51.022279 15313 leader_election.cc:302] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f4683766c12485f873f8103679e5ff5; no voters: 58e3238aaeea4f328a9eb827497df008, 738f451eb9654817824e91c33026f856
I20260929 16:28:51.022428 15709 raft_consensus.cc:2726] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:28:51.646559 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.646080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.1:60595)
0929 16:28:51.646389 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:51.646455 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":78,"threads_started":1}
I20260929 16:28:51.798511 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:51.798025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.2:35151)
0929 16:28:51.798321 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:51.798404 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":79,"threads_started":1}
I20260929 16:28:52.022313 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:52.021855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.3:37361)
0929 16:28:52.022158 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:52.022216 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":74,"threads_started":1}
I20260929 16:28:52.648154 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:52.647636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.1:35821)
0929 16:28:52.647963 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:52.648027 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":81,"threads_started":1}
I20260929 16:28:52.799834 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:52.799393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.2:34333)
0929 16:28:52.799675 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:52.799739 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":84,"threads_started":1}
I20260929 16:28:53.023801 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:53.023335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.3:53551)
0929 16:28:53.023640 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:53.023704 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":76,"threads_started":1}
I20260929 16:28:53.622923 15716 raft_consensus.cc:482] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:28:53.623098 15716 raft_consensus.cc:504] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:53.623883 15716 leader_election.cc:288] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58e3238aaeea4f328a9eb827497df008 (127.0.98.3:35689), 738f451eb9654817824e91c33026f856 (127.0.98.2:40959)
W20260929 16:28:53.624715 15315 leader_election.cc:341] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58e3238aaeea4f328a9eb827497df008 (127.0.98.3:35689): Not found: Tablet not found: 951a02fa47a74548b356c0f9d692d788
W20260929 16:28:53.624915 15313 leader_election.cc:341] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 738f451eb9654817824e91c33026f856 (127.0.98.2:40959): Not found: Tablet not found: 951a02fa47a74548b356c0f9d692d788
I20260929 16:28:53.625013 15313 leader_election.cc:302] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f4683766c12485f873f8103679e5ff5; no voters: 58e3238aaeea4f328a9eb827497df008, 738f451eb9654817824e91c33026f856
I20260929 16:28:53.625281 15716 raft_consensus.cc:2726] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:28:53.649477 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:53.648988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.1:36885)
0929 16:28:53.649307 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:53.649374 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":92,"threads_started":1}
I20260929 16:28:53.801339 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:53.800727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.2:53419)
0929 16:28:53.801012 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:53.801170 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":90,"threads_started":1}
I20260929 16:28:54.024986 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:54.024561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.3:60767)
0929 16:28:54.024839 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:54.024892 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":110,"threads_started":1}
I20260929 16:28:54.650906 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:54.650480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.1:42137)
0929 16:28:54.650743 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:54.650808 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":75,"threads_started":1}
I20260929 16:28:54.802688 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:54.802220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.2:56061)
0929 16:28:54.802525 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:54.802593 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":63,"threads_started":1}
I20260929 16:28:55.026539 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:55.025983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.3:41639)
0929 16:28:55.026357 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:55.026427 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":127,"threads_started":1}
I20260929 16:28:55.279135 15723 raft_consensus.cc:482] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:28:55.279304 15723 raft_consensus.cc:504] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:55.279726 15723 leader_election.cc:288] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58e3238aaeea4f328a9eb827497df008 (127.0.98.3:35689), 738f451eb9654817824e91c33026f856 (127.0.98.2:40959)
W20260929 16:28:55.280526 15315 leader_election.cc:341] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58e3238aaeea4f328a9eb827497df008 (127.0.98.3:35689): Not found: Tablet not found: 951a02fa47a74548b356c0f9d692d788
W20260929 16:28:55.280628 15313 leader_election.cc:341] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 738f451eb9654817824e91c33026f856 (127.0.98.2:40959): Not found: Tablet not found: 951a02fa47a74548b356c0f9d692d788
I20260929 16:28:55.280745 15313 leader_election.cc:302] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f4683766c12485f873f8103679e5ff5; no voters: 58e3238aaeea4f328a9eb827497df008, 738f451eb9654817824e91c33026f856
I20260929 16:28:55.280903 15723 raft_consensus.cc:2726] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:28:55.652386 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:55.651960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.1:55937)
0929 16:28:55.652226 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:55.652290 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
I20260929 16:28:55.804024 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:55.803588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.2:51575)
0929 16:28:55.803861 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:55.803924 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":75,"threads_started":1}
I20260929 16:28:56.027966 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:56.027484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.3:60195)
0929 16:28:56.027793 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:56.027858 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":97,"threads_started":1}
I20260929 16:28:56.653759 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:56.653283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.1:39537)
0929 16:28:56.653581 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:56.653648 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":121,"threads_started":1}
I20260929 16:28:56.805503 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:56.804959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.2:47427)
0929 16:28:56.805328 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:56.805396 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":91,"threads_started":1}
I20260929 16:28:57.029592 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:57.028974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.3:54539)
0929 16:28:57.029338 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:57.029405 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":88,"threads_started":1}
I20260929 16:28:57.392349 15730 raft_consensus.cc:482] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:28:57.392513 15730 raft_consensus.cc:504] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e3238aaeea4f328a9eb827497df008" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35689 } } peers { permanent_uuid: "9f4683766c12485f873f8103679e5ff5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35989 } } peers { permanent_uuid: "738f451eb9654817824e91c33026f856" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40959 } }
I20260929 16:28:57.393046 15730 leader_election.cc:288] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58e3238aaeea4f328a9eb827497df008 (127.0.98.3:35689), 738f451eb9654817824e91c33026f856 (127.0.98.2:40959)
W20260929 16:28:57.394042 15315 leader_election.cc:341] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58e3238aaeea4f328a9eb827497df008 (127.0.98.3:35689): Not found: Tablet not found: 951a02fa47a74548b356c0f9d692d788
W20260929 16:28:57.394142 15313 leader_election.cc:341] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 738f451eb9654817824e91c33026f856 (127.0.98.2:40959): Not found: Tablet not found: 951a02fa47a74548b356c0f9d692d788
I20260929 16:28:57.394209 15313 leader_election.cc:302] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f4683766c12485f873f8103679e5ff5; no voters: 58e3238aaeea4f328a9eb827497df008, 738f451eb9654817824e91c33026f856
I20260929 16:28:57.394353 15730 raft_consensus.cc:2726] T 951a02fa47a74548b356c0f9d692d788 P 9f4683766c12485f873f8103679e5ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:28:57.655257 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:57.654688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.1:56059)
0929 16:28:57.655091 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:57.655158 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":91,"threads_started":1}
I20260929 16:28:57.806838 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:57.806384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.2:39933)
0929 16:28:57.806684 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:57.806744 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":94,"threads_started":1}
I20260929 16:28:58.030750 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:58.030366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.3:52535)
0929 16:28:58.030604 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:58.030656 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
I20260929 16:28:58.656690 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:58.656225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.1:53415)
0929 16:28:58.656522 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:58.656587 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":85,"threads_started":1}
I20260929 16:28:58.808125 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:58.807656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.2:47169)
0929 16:28:58.807931 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:58.807990 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":67,"threads_started":1}
I20260929 16:28:59.032056 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:59.031618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.3:33005)
0929 16:28:59.031868 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:59.031930 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
I20260929 16:28:59.658257 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:59.657665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.1:37519)
0929 16:28:59.658085 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:59.658154 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":248,"threads_started":1}
I20260929 16:28:59.809587 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:28:59.809071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.2:49129)
0929 16:28:59.809421 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 16:28:59.809485 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":80,"threads_started":1}
I20260929 16:29:00.033548 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:00.032946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.3:37363)
0929 16:29:00.033382 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:00.033450 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":137,"threads_started":1}
I20260929 16:29:00.659503 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:00.659053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.1:35521)
0929 16:29:00.659339 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:00.659402 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
I20260929 16:29:00.811013 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:00.810538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.2:58657)
0929 16:29:00.810839 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:00.810910 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":77,"threads_started":1}
I20260929 16:29:01.034879 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:01.034394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.3:54947)
0929 16:29:01.034697 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:01.034782 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":92,"threads_started":1}
I20260929 16:29:01.660866 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:01.660357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.1:56951)
0929 16:29:01.660637 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:01.660754 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
I20260929 16:29:01.812393 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:01.811930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.2:34467)
0929 16:29:01.812239 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:01.812297 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":103,"threads_started":1}
I20260929 16:29:02.036238 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:02.035798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.3:38521)
0929 16:29:02.036041 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:02.036110 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":88,"threads_started":1}
I20260929 16:29:02.662225 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:02.661831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.1:38945)
0929 16:29:02.662070 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:02.662120 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":79,"threads_started":1}
I20260929 16:29:02.813944 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:02.813498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.2:60843)
0929 16:29:02.813779 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:02.813845 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":80,"threads_started":1}
I20260929 16:29:03.025825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 15262
I20260929 16:29:03.031965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 15429
I20260929 16:29:03.037714 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:03.037292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39305 (local address 127.0.98.3:37467)
0929 16:29:03.037556 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:03.037616 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":82,"threads_started":1}
I20260929 16:29:03.038511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 15563
2026-09-29T16:29:03Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19694 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:29:03 dist-test-slave-vq9v krb5kdc[15751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:29:03 dist-test-slave-vq9v krb5kdc[15751](info): set up 2 sockets
Sep 29 16:29:03 dist-test-slave-vq9v krb5kdc[15751](info): commencing operation
krb5kdc: starting...
W20260929 16:29:04.089519 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:29:04 dist-test-slave-vq9v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:29:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:29:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:29:04.224687 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39841
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:04.342528 15764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:04.342839 15764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:04.342880 15764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:04.347199 15764 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:29:04.347275 15764 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:29:04.347291 15764 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:29:04.347306 15764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:29:04.347361 15764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:04.347380 15764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:04.347455 15764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:29:04.347478 15764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:29:04.353130 15764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:29:04.354430 15764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:04.355536 15764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:29:04.364559 15764 server_base.cc:1034] running on GCE node
W20260929 16:29:04.364507 15772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:04.364507 15769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:04.364511 15770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:04.365535 15764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:04.366778 15764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:04.367949 15764 hybrid_clock.cc:648] HybridClock initialized: now 1790699344367938 us; error 34 us; skew 500 ppm
Sep 29 16:29:04 dist-test-slave-vq9v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:04.378564 15764 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:29:04.379591 15764 webserver.cc:460] Webserver started at http://127.0.98.62:42755/ using document root <none> and password file <none>
I20260929 16:29:04.380170 15764 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:04.380240 15764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:04.380422 15764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:04.382710 15764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "81ee0667256042b6834b2b4b596ed0e1"
format_stamp: "Formatted at 2026-09-29 16:29:04 on dist-test-slave-vq9v"
server_key: "d2c058f5a5b5d02335cd8370a835f257"
server_key_iv: "825d1e2722c2969dfb690c484728449e"
server_key_version: "encryptionkey@0"
I20260929 16:29:04.383260 15764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "81ee0667256042b6834b2b4b596ed0e1"
format_stamp: "Formatted at 2026-09-29 16:29:04 on dist-test-slave-vq9v"
server_key: "d2c058f5a5b5d02335cd8370a835f257"
server_key_iv: "825d1e2722c2969dfb690c484728449e"
server_key_version: "encryptionkey@0"
I20260929 16:29:04.387216 15764 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:29:04.389912 15779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:04.391289 15764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:29:04.391446 15764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "81ee0667256042b6834b2b4b596ed0e1"
format_stamp: "Formatted at 2026-09-29 16:29:04 on dist-test-slave-vq9v"
server_key: "d2c058f5a5b5d02335cd8370a835f257"
server_key_iv: "825d1e2722c2969dfb690c484728449e"
server_key_version: "encryptionkey@0"
I20260929 16:29:04.391564 15764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:04.404640 15764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:04.407907 15764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:04.408064 15764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:04.415795 15764 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39841
I20260929 16:29:04.416181 15831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39841 every 8 connection(s)
I20260929 16:29:04.416872 15764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:29:04.419890 15832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:04.421924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 15764
I20260929 16:29:04.422067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:29:04.422399 392 external_mini_cluster.cc:1428] Setting key f8ea72df8f9ffa091fe7a95a821fd87d
I20260929 16:29:04.426258 15832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1: Bootstrap starting.
I20260929 16:29:04.428594 15832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:04.429476 15832 log.cc:824] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:29:04 dist-test-slave-vq9v krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:29:04.431604 15832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1: No bootstrap required, opened a new log
I20260929 16:29:04.435601 15832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ee0667256042b6834b2b4b596ed0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } }
I20260929 16:29:04.435875 15832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:04.435957 15832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81ee0667256042b6834b2b4b596ed0e1, State: Initialized, Role: FOLLOWER
I20260929 16:29:04.436484 15832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ee0667256042b6834b2b4b596ed0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } }
I20260929 16:29:04.436663 15832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:29:04.436759 15832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:29:04.436882 15832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:04.437294 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:04.423829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52414 (local address 127.0.98.62:39841)
0929 16:29:04.424318 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:04.424334 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:04.424371 (+ 37us) server_negotiation.cc:407] Connection header received
0929 16:29:04.425331 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:04.425353 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:04.425723 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:04.426098 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:04.426939 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:04.427955 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:04.428843 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:04.429422 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:04.429596 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:04.432189 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:04.432194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:04.432209 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:04.432237 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:04.434057 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:04.434572 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:04.434577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:04.434582 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:04.434698 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:04.435020 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:04.435025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:04.435027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:04.435455 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:04.435673 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:04.435972 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 16:29:04.436326 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":147,"threads_started":1}
I20260929 16:29:04.438237 15832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ee0667256042b6834b2b4b596ed0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } }
I20260929 16:29:04.438617 15832 leader_election.cc:302] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81ee0667256042b6834b2b4b596ed0e1; no voters:
I20260929 16:29:04.438956 15832 leader_election.cc:288] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:29:04.439106 15837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:04.439316 15837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 1 LEADER]: Becoming Leader. State: Replica: 81ee0667256042b6834b2b4b596ed0e1, State: Running, Role: LEADER
I20260929 16:29:04.439688 15837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ee0667256042b6834b2b4b596ed0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } }
I20260929 16:29:04.440313 15832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:29:04.440961 15837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81ee0667256042b6834b2b4b596ed0e1. Latest consensus state: current_term: 1 leader_uuid: "81ee0667256042b6834b2b4b596ed0e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ee0667256042b6834b2b4b596ed0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } }
I20260929 16:29:04.441118 15837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:04.441498 15838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81ee0667256042b6834b2b4b596ed0e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ee0667256042b6834b2b4b596ed0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } }
I20260929 16:29:04.441603 15844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:29:04.441686 15838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:04.444866 15844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:29:04.450664 15844 catalog_manager.cc:1347] Generated new cluster ID: c1a93fd2a69442d5b83d311a9f4599a9
I20260929 16:29:04.450773 15844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:29:04.458877 15844 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:29:04.459892 15844 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:29:04.463992 15844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1: Generated new TSK 0
I20260929 16:29:04.464725 15844 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 16:29:04.497795 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39841
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:04.611819 15858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:04.612095 15858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:04.612149 15858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:04.616451 15858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:04.616528 15858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:04.616592 15858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:29:04.622179 15858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:29:04.623368 15858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:04.624491 15858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:04.634503 15864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:04.634507 15863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:04.634507 15866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:04.634760 15858 server_base.cc:1034] running on GCE node
I20260929 16:29:04.635236 15858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:04.635897 15858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:04.637115 15858 hybrid_clock.cc:648] HybridClock initialized: now 1790699344637099 us; error 34 us; skew 500 ppm
Sep 29 16:29:04 dist-test-slave-vq9v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:04.639914 15858 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:29:04.640815 15858 webserver.cc:460] Webserver started at http://127.0.98.1:37137/ using document root <none> and password file <none>
I20260929 16:29:04.641466 15858 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:04.641577 15858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:04.641845 15858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:04.643919 15858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "31ed5e5c9cc0450783fa75521dff94e4"
format_stamp: "Formatted at 2026-09-29 16:29:04 on dist-test-slave-vq9v"
server_key: "88138fa4052858362669b96776625c4e"
server_key_iv: "8b35f320572a016105532f2d4d834bc7"
server_key_version: "encryptionkey@0"
I20260929 16:29:04.644482 15858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "31ed5e5c9cc0450783fa75521dff94e4"
format_stamp: "Formatted at 2026-09-29 16:29:04 on dist-test-slave-vq9v"
server_key: "88138fa4052858362669b96776625c4e"
server_key_iv: "8b35f320572a016105532f2d4d834bc7"
server_key_version: "encryptionkey@0"
I20260929 16:29:04.648249 15858 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 16:29:04.650782 15873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:04.652024 15858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:29:04.652165 15858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "31ed5e5c9cc0450783fa75521dff94e4"
format_stamp: "Formatted at 2026-09-29 16:29:04 on dist-test-slave-vq9v"
server_key: "88138fa4052858362669b96776625c4e"
server_key_iv: "8b35f320572a016105532f2d4d834bc7"
server_key_version: "encryptionkey@0"
I20260929 16:29:04.652283 15858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:04.667084 15858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:04.670308 15858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:04.670531 15858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:04.671146 15858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:04.671999 15858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:04.672071 15858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:04.672140 15858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:04.672192 15858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:04.681604 15858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46609
I20260929 16:29:04.681633 15986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46609 every 8 connection(s)
I20260929 16:29:04.682508 15858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:29:04.684239 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 15858
I20260929 16:29:04.684376 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:29:04.684661 392 external_mini_cluster.cc:1428] Setting key a239a58e2f02721c0c43934d5c487664
Sep 29 16:29:04 dist-test-slave-vq9v krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 16:29:04.698485 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:04.684366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35303 (local address 127.0.98.62:39841)
0929 16:29:04.684501 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:04.684506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:04.685517 (+ 1011us) server_negotiation.cc:407] Connection header received
0929 16:29:04.687247 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:04.687249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:04.687290 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:04.687359 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:04.688755 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:04.689698 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:04.690967 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:04.691430 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:04.691565 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:04.694237 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:04.694241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:04.694246 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:04.694283 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:04.695978 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:04.696619 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:04.696623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:04.696625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:04.696725 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:04.697192 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:04.697220 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:04.697224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:04.697483 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:04.697575 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:04.698244 (+ 669us) server_negotiation.cc:299] Negotiation successful
0929 16:29:04.698357 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:29:04.699440 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:04.684798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.1:35303)
0929 16:29:04.685347 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:04.685408 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:04.687004 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:04.687539 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:04.687551 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:04.688096 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:04.688597 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:04.688610 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:04.689842 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:04.689850 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:04.690825 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:04.690835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:04.691605 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:04.691608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:04.691751 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:04.692376 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:04.692392 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:04.694050 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:04.696162 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:04.696170 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:04.696187 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:04.696500 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:04.696867 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:04.696871 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:04.696873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:04.697095 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:04.697619 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:04.697628 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:04.697969 (+ 341us) client_negotiation.cc:769] Sending connection context
0929 16:29:04.698191 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 16:29:04.698574 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":129,"threads_started":1}
I20260929 16:29:04.700872 15987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39841
I20260929 16:29:04.701174 15987 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:04.701869 15987 heartbeater.cc:511] Master 127.0.98.62:39841 requested a full tablet report, sending...
I20260929 16:29:04.703627 15796 ts_manager.cc:194] Registered new tserver with Master: 31ed5e5c9cc0450783fa75521dff94e4 (127.0.98.1:46609)
I20260929 16:29:04.704885 15796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 16:29:04.716722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39841
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:04.831856 15992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:04.832173 15992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:04.832254 15992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:04.836426 15992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:04.836611 15992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:04.836782 15992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:29:04.842840 15992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:29:04.844012 15992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:04.845158 15992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:04.855468 16000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:04.855701 15992 server_base.cc:1034] running on GCE node
W20260929 16:29:04.855454 15997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:04.855454 15998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:04.856292 15992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:04.857074 15992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:04.858285 15992 hybrid_clock.cc:648] HybridClock initialized: now 1790699344858273 us; error 35 us; skew 500 ppm
Sep 29 16:29:04 dist-test-slave-vq9v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:04.861246 15992 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:29:04.862221 15992 webserver.cc:460] Webserver started at http://127.0.98.2:42221/ using document root <none> and password file <none>
I20260929 16:29:04.862809 15992 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:04.862906 15992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:04.863143 15992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:04.865248 15992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "28c94367cb9e4c69b932928e198b4fb2"
format_stamp: "Formatted at 2026-09-29 16:29:04 on dist-test-slave-vq9v"
server_key: "8ac47c2d2b67ddf272aa5afaeb9aa05d"
server_key_iv: "60957d64c935970ae8ced538997ede1f"
server_key_version: "encryptionkey@0"
I20260929 16:29:04.865852 15992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "28c94367cb9e4c69b932928e198b4fb2"
format_stamp: "Formatted at 2026-09-29 16:29:04 on dist-test-slave-vq9v"
server_key: "8ac47c2d2b67ddf272aa5afaeb9aa05d"
server_key_iv: "60957d64c935970ae8ced538997ede1f"
server_key_version: "encryptionkey@0"
I20260929 16:29:04.869803 15992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 16:29:04.872452 16007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:04.873840 15992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:29:04.873978 15992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "28c94367cb9e4c69b932928e198b4fb2"
format_stamp: "Formatted at 2026-09-29 16:29:04 on dist-test-slave-vq9v"
server_key: "8ac47c2d2b67ddf272aa5afaeb9aa05d"
server_key_iv: "60957d64c935970ae8ced538997ede1f"
server_key_version: "encryptionkey@0"
I20260929 16:29:04.874104 15992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:04.889449 15992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:04.892596 15992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:04.892824 15992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:04.893548 15992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:04.894419 15992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:04.894492 15992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:04.894563 15992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:04.894614 15992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:04.903975 15992 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37725
I20260929 16:29:04.904002 16120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37725 every 8 connection(s)
I20260929 16:29:04.905050 15992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:29:04.913859 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 15992
I20260929 16:29:04.913997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:29:04.914379 392 external_mini_cluster.cc:1428] Setting key a0ee5607014df7d8588070d0c1b08a77
Sep 29 16:29:04 dist-test-slave-vq9v krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:29:04.919281 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:04.906901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46445 (local address 127.0.98.62:39841)
0929 16:29:04.907035 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:04.907038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:04.907852 (+ 814us) server_negotiation.cc:407] Connection header received
0929 16:29:04.909060 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:04.909063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:04.909112 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:04.909188 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:04.910453 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:04.910987 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:04.912202 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:04.912433 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:04.912530 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:04.915455 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:04.915459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:04.915461 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:04.915484 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:04.916993 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:04.917624 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:04.917629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:04.917631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:04.917717 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:04.918102 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:04.918106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:04.918107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:04.918308 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:04.918418 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:04.918958 (+ 540us) server_negotiation.cc:299] Negotiation successful
0929 16:29:04.919107 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:29:04.920143 16123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:04.907239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.2:46445)
0929 16:29:04.907685 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:04.907727 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:04.908860 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:04.909396 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:04.909403 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:04.909755 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:04.910305 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:04.910324 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:04.911148 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:04.911151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:04.912077 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:04.912087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:04.912595 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:04.912599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:04.912760 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:04.913424 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:04.913441 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:04.915294 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:04.917121 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:04.917128 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:04.917146 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:04.917512 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:04.917825 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:04.917828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:04.917830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:04.917992 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:04.918418 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:04.918426 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:04.918705 (+ 279us) client_negotiation.cc:769] Sending connection context
0929 16:29:04.918902 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 16:29:04.919265 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":103,"threads_started":1}
I20260929 16:29:04.921469 16121 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39841
I20260929 16:29:04.921729 16121 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:04.922254 16121 heartbeater.cc:511] Master 127.0.98.62:39841 requested a full tablet report, sending...
I20260929 16:29:04.923401 15796 ts_manager.cc:194] Registered new tserver with Master: 28c94367cb9e4c69b932928e198b4fb2 (127.0.98.2:37725)
I20260929 16:29:04.924311 15796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46445
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 16:29:04.945323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39841
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:05.066473 16126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:05.066758 16126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:05.066802 16126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:05.070885 16126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:05.070955 16126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:05.071014 16126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:29:05.076761 16126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:29:05.078016 16126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:05.079182 16126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:05.089600 16131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:05.089593 16132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:05.089636 16134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:05.089922 16126 server_base.cc:1034] running on GCE node
I20260929 16:29:05.090468 16126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:05.091135 16126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:05.092368 16126 hybrid_clock.cc:648] HybridClock initialized: now 1790699345092333 us; error 47 us; skew 500 ppm
Sep 29 16:29:05 dist-test-slave-vq9v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:05.095780 16126 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:29:05.096971 16126 webserver.cc:460] Webserver started at http://127.0.98.3:43989/ using document root <none> and password file <none>
I20260929 16:29:05.097697 16126 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:05.097801 16126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:05.098062 16126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:29:05.100390 16126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "2e1c0eed17954a1cac7908f34e3d8b63"
format_stamp: "Formatted at 2026-09-29 16:29:05 on dist-test-slave-vq9v"
server_key: "b90ac823f93da59a388fecc6ff796269"
server_key_iv: "648534cbe3d0968e319710ba4ee21f0f"
server_key_version: "encryptionkey@0"
I20260929 16:29:05.101032 16126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2e1c0eed17954a1cac7908f34e3d8b63"
format_stamp: "Formatted at 2026-09-29 16:29:05 on dist-test-slave-vq9v"
server_key: "b90ac823f93da59a388fecc6ff796269"
server_key_iv: "648534cbe3d0968e319710ba4ee21f0f"
server_key_version: "encryptionkey@0"
I20260929 16:29:05.105384 16126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:29:05.107980 16141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:05.109438 16126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:29:05.109598 16126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "2e1c0eed17954a1cac7908f34e3d8b63"
format_stamp: "Formatted at 2026-09-29 16:29:05 on dist-test-slave-vq9v"
server_key: "b90ac823f93da59a388fecc6ff796269"
server_key_iv: "648534cbe3d0968e319710ba4ee21f0f"
server_key_version: "encryptionkey@0"
I20260929 16:29:05.109746 16126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:05.123396 16126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:05.127417 16126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:05.127678 16126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:05.128351 16126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:05.129362 16126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:05.129437 16126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:05.129518 16126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:05.129587 16126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:05.139829 16126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37331
I20260929 16:29:05.139846 16254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37331 every 8 connection(s)
I20260929 16:29:05.140914 16126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:29:05.142630 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 16126
I20260929 16:29:05.142804 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:29:05.143150 392 external_mini_cluster.cc:1428] Setting key 9320e209d3178fb012a5c6ecd5534843
Sep 29 16:29:05 dist-test-slave-vq9v krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:29:05.156620 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:05.142927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37411 (local address 127.0.98.62:39841)
0929 16:29:05.143102 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:05.143108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:05.144241 (+ 1133us) server_negotiation.cc:407] Connection header received
0929 16:29:05.145987 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:05.145990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:05.146035 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:05.146111 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:05.147412 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.147933 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.149356 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.149565 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.149646 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:05.152373 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:05.152376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:05.152378 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:05.152404 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:05.154121 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:05.154708 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:05.154712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:05.154713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:05.154800 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:05.155173 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:05.155208 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:05.155213 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:05.155470 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:05.155579 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:05.156212 (+ 633us) server_negotiation.cc:299] Negotiation successful
0929 16:29:05.156422 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:29:05.157711 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:05.143355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.3:37411)
0929 16:29:05.144013 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:05.144142 (+ 129us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:05.145776 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:05.146290 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:05.146298 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:05.146691 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:05.147272 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:05.147288 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.148133 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.148136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:05.149149 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:05.149159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.149709 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.149712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:05.149870 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:05.150615 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:05.150632 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:05.152168 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:05.154242 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:05.154247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:05.154276 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:05.154562 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:05.154905 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:05.154907 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:05.154908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:05.155051 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:05.155591 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:05.155597 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:05.155901 (+ 304us) client_negotiation.cc:769] Sending connection context
0929 16:29:05.156128 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 16:29:05.156605 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":152,"threads_started":1}
I20260929 16:29:05.158874 16255 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39841
I20260929 16:29:05.159097 16255 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:05.159559 16255 heartbeater.cc:511] Master 127.0.98.62:39841 requested a full tablet report, sending...
I20260929 16:29:05.160595 15796 ts_manager.cc:194] Registered new tserver with Master: 2e1c0eed17954a1cac7908f34e3d8b63 (127.0.98.3:37331)
I20260929 16:29:05.161365 15796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37411
I20260929 16:29:05.169867 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:29:05.179442 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:05.171979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.0.98.62:39841)
0929 16:29:05.172124 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:05.172129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:05.172413 (+ 284us) server_negotiation.cc:407] Connection header received
0929 16:29:05.172487 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:05.172489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:05.172533 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:05.172633 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:05.173327 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.173860 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.174660 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.174874 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.174957 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:05.176296 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:05.176299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:05.176301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:05.176331 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:05.177789 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:05.178209 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:05.178212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:05.178213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:05.178315 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:05.178620 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:05.178622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:05.178624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:05.178851 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:05.178962 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:05.179163 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 16:29:05.179260 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:29:05.182580 15796 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:29:05.185817 15796 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:29:05.201344 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:05.195410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52108 (local address 127.0.98.3:37331)
0929 16:29:05.196644 (+ 1234us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:05.196658 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:05.196684 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:29:05.196768 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:05.196773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:05.197060 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:05.197264 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:05.197595 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.198406 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.200020 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.200609 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.200738 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:05.200894 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:05.201056 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 16:29:05.201160 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1131,"thread_start_us":144,"threads_started":1}
I20260929 16:29:05.201335 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:05.195132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37331 (local address 127.0.0.1:52108)
0929 16:29:05.196091 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:05.196156 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:05.196323 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:05.197268 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:05.197274 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:05.197297 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:05.197454 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:05.197461 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.198540 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.198546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:05.199837 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:05.199887 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.200737 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.200741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:05.200844 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:05.200880 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:29:05.201026 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 16:29:05.201162 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":28,"thread_start_us":119,"threads_started":1}
I20260929 16:29:05.202387 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:05.195245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37725 (local address 127.0.0.1:40274)
0929 16:29:05.196091 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:05.196156 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:05.196323 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:05.197475 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:05.197478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:05.197496 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:05.197597 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:05.197602 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.199234 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.199238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:05.200763 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:05.200775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.201958 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.201962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:05.202054 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:05.202095 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 16:29:05.202187 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 16:29:05.202255 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":59,"spinlock_wait_cycles":1920,"thread_start_us":117,"threads_started":1}
I20260929 16:29:05.202492 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:05.195818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.0.98.2:37725)
0929 16:29:05.196644 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:05.196658 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:05.196684 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:29:05.196768 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:05.196773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:05.197068 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:05.197264 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:05.197727 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.199055 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.200920 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.201839 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.201962 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:05.202107 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:05.202217 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 16:29:05.202310 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":109,"threads_started":1}
I20260929 16:29:05.202647 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:05.195596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46609 (local address 127.0.0.1:44226)
0929 16:29:05.196576 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:05.196615 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:05.196702 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:05.197450 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:05.197453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:05.197467 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:05.197595 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:05.197602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.199234 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.199238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:05.200765 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:05.200775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.202196 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.202200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:05.202297 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:05.202321 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:29:05.202400 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 16:29:05.202468 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"mutex_wait_us":164,"spinlock_wait_cycles":768,"thread_start_us":85,"threads_started":1}
I20260929 16:29:05.203122 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:05.196047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44226 (local address 127.0.98.1:46609)
0929 16:29:05.196444 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:05.196460 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:05.196808 (+ 348us) server_negotiation.cc:407] Connection header received
0929 16:29:05.196904 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:05.196911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:05.197108 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:05.197298 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:05.197783 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.199063 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.200921 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.201839 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.201962 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:05.202108 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:05.202920 (+ 812us) server_negotiation.cc:299] Negotiation successful
0929 16:29:05.203002 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":64,"threads_started":1}
I20260929 16:29:05.204949 16189 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab89fca00524db0980f8523ff49635d (DEFAULT_TABLE table=test-table [id=df37e69492664bb5b1a5db817f7efa22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:05.205674 16055 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab89fca00524db0980f8523ff49635d (DEFAULT_TABLE table=test-table [id=df37e69492664bb5b1a5db817f7efa22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:05.206286 16189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab89fca00524db0980f8523ff49635d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:05.206130 15921 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab89fca00524db0980f8523ff49635d (DEFAULT_TABLE table=test-table [id=df37e69492664bb5b1a5db817f7efa22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:05.206704 16055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab89fca00524db0980f8523ff49635d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:05.207057 15921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab89fca00524db0980f8523ff49635d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:05.213837 16274 tablet_bootstrap.cc:492] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63: Bootstrap starting.
I20260929 16:29:05.214658 16273 tablet_bootstrap.cc:492] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4: Bootstrap starting.
I20260929 16:29:05.216189 16274 tablet_bootstrap.cc:654] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:05.216423 16273 tablet_bootstrap.cc:654] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:05.217135 16273 log.cc:824] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:05.217381 16274 log.cc:824] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:05.217631 16272 tablet_bootstrap.cc:492] T 8ab89fca00524db0980f8523ff49635d P 28c94367cb9e4c69b932928e198b4fb2: Bootstrap starting.
I20260929 16:29:05.219310 16273 tablet_bootstrap.cc:492] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4: No bootstrap required, opened a new log
I20260929 16:29:05.219513 16273 ts_tablet_manager.cc:1397] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 16:29:05.219774 16274 tablet_bootstrap.cc:492] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63: No bootstrap required, opened a new log
I20260929 16:29:05.219964 16274 ts_tablet_manager.cc:1397] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 16:29:05.220465 16272 tablet_bootstrap.cc:654] T 8ab89fca00524db0980f8523ff49635d P 28c94367cb9e4c69b932928e198b4fb2: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:05.221290 16272 log.cc:824] T 8ab89fca00524db0980f8523ff49635d P 28c94367cb9e4c69b932928e198b4fb2: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:05.223089 16272 tablet_bootstrap.cc:492] T 8ab89fca00524db0980f8523ff49635d P 28c94367cb9e4c69b932928e198b4fb2: No bootstrap required, opened a new log
I20260929 16:29:05.223308 16272 ts_tablet_manager.cc:1397] T 8ab89fca00524db0980f8523ff49635d P 28c94367cb9e4c69b932928e198b4fb2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:29:05.224022 16274 raft_consensus.cc:348] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:05.224022 16273 raft_consensus.cc:348] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:05.224303 16273 raft_consensus.cc:374] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:05.224303 16274 raft_consensus.cc:374] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:05.224365 16274 raft_consensus.cc:729] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1c0eed17954a1cac7908f34e3d8b63, State: Initialized, Role: FOLLOWER
I20260929 16:29:05.224393 16273 raft_consensus.cc:729] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31ed5e5c9cc0450783fa75521dff94e4, State: Initialized, Role: FOLLOWER
I20260929 16:29:05.224879 16273 consensus_queue.cc:260] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:05.224879 16274 consensus_queue.cc:260] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:05.226027 16274 ts_tablet_manager.cc:1428] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260929 16:29:05.226042 16273 ts_tablet_manager.cc:1428] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260929 16:29:05.226418 16255 heartbeater.cc:503] Master 127.0.98.62:39841 was elected leader, sending a full tablet report...
I20260929 16:29:05.226418 15987 heartbeater.cc:503] Master 127.0.98.62:39841 was elected leader, sending a full tablet report...
I20260929 16:29:05.227077 16272 raft_consensus.cc:348] T 8ab89fca00524db0980f8523ff49635d P 28c94367cb9e4c69b932928e198b4fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:05.227322 16272 raft_consensus.cc:374] T 8ab89fca00524db0980f8523ff49635d P 28c94367cb9e4c69b932928e198b4fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:05.227445 16272 raft_consensus.cc:729] T 8ab89fca00524db0980f8523ff49635d P 28c94367cb9e4c69b932928e198b4fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28c94367cb9e4c69b932928e198b4fb2, State: Initialized, Role: FOLLOWER
I20260929 16:29:05.227983 16272 consensus_queue.cc:260] T 8ab89fca00524db0980f8523ff49635d P 28c94367cb9e4c69b932928e198b4fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:05.228677 16121 heartbeater.cc:503] Master 127.0.98.62:39841 was elected leader, sending a full tablet report...
I20260929 16:29:05.228910 16272 ts_tablet_manager.cc:1428] T 8ab89fca00524db0980f8523ff49635d P 28c94367cb9e4c69b932928e198b4fb2: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260929 16:29:05.302235 16278 raft_consensus.cc:482] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:29:05.302485 16278 raft_consensus.cc:504] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:05.303793 16278 leader_election.cc:288] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28c94367cb9e4c69b932928e198b4fb2 (127.0.98.2:37725), 31ed5e5c9cc0450783fa75521dff94e4 (127.0.98.1:46609)
I20260929 16:29:05.307791 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:05.304062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37725 (local address 127.0.98.3:52809)
0929 16:29:05.304185 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:05.304202 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:05.304306 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:05.304585 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:05.304588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:05.304606 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:05.304739 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:05.304745 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.305708 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.305714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:05.306561 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:05.306572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.307296 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.307300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:05.307427 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:05.307449 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:29:05.307551 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 16:29:05.307639 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 16:29:05.307791 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:05.304178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52809 (local address 127.0.98.2:37725)
0929 16:29:05.304320 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:05.304326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:05.304344 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:29:05.304407 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:05.304412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:05.304467 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:05.304579 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:05.304886 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.305586 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.306712 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.307157 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.307293 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:05.307372 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:05.307571 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 16:29:05.307654 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:29:05.307989 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:05.304069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46609 (local address 127.0.98.3:51067)
0929 16:29:05.304542 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:05.304554 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:05.304683 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:05.304958 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:05.304961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:05.304976 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:05.305099 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:05.305105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.306010 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.306016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:05.306839 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:05.306848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.307528 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.307532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:05.307642 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:05.307669 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:29:05.307768 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 16:29:05.307840 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":112,"thread_start_us":182,"threads_started":1}
I20260929 16:29:05.307972 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:05.304178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51067 (local address 127.0.98.1:46609)
0929 16:29:05.304309 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:05.304315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:05.304645 (+ 330us) server_negotiation.cc:407] Connection header received
0929 16:29:05.304808 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:05.304811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:05.304863 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:05.304948 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:05.305244 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.305891 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.306977 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.307400 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.307523 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:05.307587 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:05.307778 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 16:29:05.307854 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:29:05.308714 15941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab89fca00524db0980f8523ff49635d" candidate_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31ed5e5c9cc0450783fa75521dff94e4" is_pre_election: true
I20260929 16:29:05.308723 16075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab89fca00524db0980f8523ff49635d" candidate_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28c94367cb9e4c69b932928e198b4fb2" is_pre_election: true
I20260929 16:29:05.309024 16075 raft_consensus.cc:2454] T 8ab89fca00524db0980f8523ff49635d P 28c94367cb9e4c69b932928e198b4fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e1c0eed17954a1cac7908f34e3d8b63 in term 0.
I20260929 16:29:05.309026 15941 raft_consensus.cc:2454] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e1c0eed17954a1cac7908f34e3d8b63 in term 0.
I20260929 16:29:05.309630 16144 leader_election.cc:302] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28c94367cb9e4c69b932928e198b4fb2, 2e1c0eed17954a1cac7908f34e3d8b63, 31ed5e5c9cc0450783fa75521dff94e4; no voters:
I20260929 16:29:05.309904 16278 raft_consensus.cc:2781] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:29:05.310011 16278 raft_consensus.cc:482] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:29:05.310082 16278 raft_consensus.cc:3037] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:05.311206 16278 raft_consensus.cc:504] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:05.311618 16278 leader_election.cc:288] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63 [CANDIDATE]: Term 1 election: Requested vote from peers 28c94367cb9e4c69b932928e198b4fb2 (127.0.98.2:37725), 31ed5e5c9cc0450783fa75521dff94e4 (127.0.98.1:46609)
I20260929 16:29:05.312078 16075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab89fca00524db0980f8523ff49635d" candidate_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28c94367cb9e4c69b932928e198b4fb2"
I20260929 16:29:05.312078 15941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab89fca00524db0980f8523ff49635d" candidate_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31ed5e5c9cc0450783fa75521dff94e4"
I20260929 16:29:05.312227 16075 raft_consensus.cc:3037] T 8ab89fca00524db0980f8523ff49635d P 28c94367cb9e4c69b932928e198b4fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:05.312227 15941 raft_consensus.cc:3037] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:05.313652 15941 raft_consensus.cc:2454] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e1c0eed17954a1cac7908f34e3d8b63 in term 1.
I20260929 16:29:05.313671 16075 raft_consensus.cc:2454] T 8ab89fca00524db0980f8523ff49635d P 28c94367cb9e4c69b932928e198b4fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e1c0eed17954a1cac7908f34e3d8b63 in term 1.
I20260929 16:29:05.314031 16145 leader_election.cc:302] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e1c0eed17954a1cac7908f34e3d8b63, 31ed5e5c9cc0450783fa75521dff94e4; no voters:
I20260929 16:29:05.314347 16278 raft_consensus.cc:2781] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:05.314627 16278 raft_consensus.cc:686] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63 [term 1 LEADER]: Becoming Leader. State: Replica: 2e1c0eed17954a1cac7908f34e3d8b63, State: Running, Role: LEADER
I20260929 16:29:05.314975 16278 consensus_queue.cc:237] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:05.318368 15795 catalog_manager.cc:5495] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e1c0eed17954a1cac7908f34e3d8b63 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } health_report { overall_health: UNKNOWN } } }
I20260929 16:29:05.356488 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:05.352741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52112 (local address 127.0.98.3:37331)
0929 16:29:05.352907 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:05.352914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:05.352932 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:29:05.353029 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:05.353031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:05.353087 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:05.353205 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:29:05.353488 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.354005 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.354880 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:05.355104 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:05.355218 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:05.355533 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:29:05.356008 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:29:05.356172 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:05.356293 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 16:29:05.356352 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 16:29:05.362766 15941 raft_consensus.cc:1261] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Refusing update from remote peer 2e1c0eed17954a1cac7908f34e3d8b63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:29:05.362766 16075 raft_consensus.cc:1261] T 8ab89fca00524db0980f8523ff49635d P 28c94367cb9e4c69b932928e198b4fb2 [term 1 FOLLOWER]: Refusing update from remote peer 2e1c0eed17954a1cac7908f34e3d8b63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:29:05.363512 16278 consensus_queue.cc:1035] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:29:05.363749 16282 consensus_queue.cc:1035] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:29:05.371968 16287 mvcc.cc:205] Tried to move back new op lower bound from 7334704518597443584 to 7334704518412783616. Current Snapshot: MvccSnapshot[applied={T|T < 7334704518597443584}]
I20260929 16:29:05.373463 16286 mvcc.cc:205] Tried to move back new op lower bound from 7334704518597443584 to 7334704518412783616. Current Snapshot: MvccSnapshot[applied={T|T < 7334704518597443584}]
I20260929 16:29:05.380264 15795 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:29:05.380425 15795 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:29:05.382967 15795 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1: Sending DeleteTablet for 3 replicas of tablet 8ab89fca00524db0980f8523ff49635d
I20260929 16:29:05.383816 16055 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ab89fca00524db0980f8523ff49635d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40274
I20260929 16:29:05.384147 15921 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ab89fca00524db0980f8523ff49635d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44226
I20260929 16:29:05.384347 16294 tablet_replica.cc:330] T 8ab89fca00524db0980f8523ff49635d P 28c94367cb9e4c69b932928e198b4fb2: stopping tablet replica
I20260929 16:29:05.384334 16189 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ab89fca00524db0980f8523ff49635d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52108
I20260929 16:29:05.384688 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 15858
I20260929 16:29:05.384696 16294 raft_consensus.cc:2229] T 8ab89fca00524db0980f8523ff49635d P 28c94367cb9e4c69b932928e198b4fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:29:05.384989 16296 tablet_replica.cc:330] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63: stopping tablet replica
I20260929 16:29:05.385018 16294 raft_consensus.cc:2258] T 8ab89fca00524db0980f8523ff49635d P 28c94367cb9e4c69b932928e198b4fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:05.385380 16296 raft_consensus.cc:2229] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:29:05.385787 16296 raft_consensus.cc:2258] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:05.387331 16296 ts_tablet_manager.cc:1905] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:05.387331 16294 ts_tablet_manager.cc:1905] T 8ab89fca00524db0980f8523ff49635d P 28c94367cb9e4c69b932928e198b4fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:05.390254 16296 ts_tablet_manager.cc:1918] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:29:05.390436 16296 log.cc:1196] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/8ab89fca00524db0980f8523ff49635d
I20260929 16:29:05.390843 16296 ts_tablet_manager.cc:1939] T 8ab89fca00524db0980f8523ff49635d P 2e1c0eed17954a1cac7908f34e3d8b63: Deleting consensus metadata
I20260929 16:29:05.390890 16294 ts_tablet_manager.cc:1918] T 8ab89fca00524db0980f8523ff49635d P 28c94367cb9e4c69b932928e198b4fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:29:05.391077 16294 log.cc:1196] T 8ab89fca00524db0980f8523ff49635d P 28c94367cb9e4c69b932928e198b4fb2: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/8ab89fca00524db0980f8523ff49635d
I20260929 16:29:05.391472 16294 ts_tablet_manager.cc:1939] T 8ab89fca00524db0980f8523ff49635d P 28c94367cb9e4c69b932928e198b4fb2: Deleting consensus metadata
I20260929 16:29:05.392194 15781 catalog_manager.cc:4841] TS 2e1c0eed17954a1cac7908f34e3d8b63 (127.0.98.3:37331): tablet 8ab89fca00524db0980f8523ff49635d (table test-table [id=df37e69492664bb5b1a5db817f7efa22]) successfully deleted
I20260929 16:29:05.392731 15782 catalog_manager.cc:4841] TS 28c94367cb9e4c69b932928e198b4fb2 (127.0.98.2:37725): tablet 8ab89fca00524db0980f8523ff49635d (table test-table [id=df37e69492664bb5b1a5db817f7efa22]) successfully deleted
I20260929 16:29:05.393447 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 15992
W20260929 16:29:05.393442 15783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46609 (error 108)
I20260929 16:29:05.398365 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:05.397955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46609 (local address 127.0.0.1:44232)
0929 16:29:05.398149 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:05.398247 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 16:29:05.398586 15783 catalog_manager.cc:4584] TS 31ed5e5c9cc0450783fa75521dff94e4 (127.0.98.1:46609): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ab89fca00524db0980f8523ff49635d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46609: connect: Connection refused (error 111)
I20260929 16:29:05.400192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 16126
I20260929 16:29:05.405460 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 15764
I20260929 16:29:25.412945 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39841
--webserver_interface=127.0.98.62
--webserver_port=42755
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:25.530028 16298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:25.530356 16298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:25.530449 16298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:25.534677 16298 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:29:25.534772 16298 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:29:25.534802 16298 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:29:25.534833 16298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:29:25.534874 16298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:25.534917 16298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:25.534953 16298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:29:25.534986 16298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:29:25.540539 16298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:29:25.541733 16298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:25.542771 16298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:25.556413 16305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:25.556507 16298 server_base.cc:1034] running on GCE node
W20260929 16:29:25.556413 16304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:25.556461 16307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:25.557367 16298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:25.558571 16298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:25.559796 16298 hybrid_clock.cc:648] HybridClock initialized: now 1790699365559760 us; error 50 us; skew 500 ppm
Sep 29 16:29:25 dist-test-slave-vq9v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:25.567847 16298 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:29:25.568859 16298 webserver.cc:460] Webserver started at http://127.0.98.62:42755/ using document root <none> and password file <none>
I20260929 16:29:25.569578 16298 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:25.569659 16298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:25.574030 16298 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:29:25.576359 16314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:25.577814 16298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:29:25.577971 16298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "81ee0667256042b6834b2b4b596ed0e1"
format_stamp: "Formatted at 2026-09-29 16:29:04 on dist-test-slave-vq9v"
server_key: "d2c058f5a5b5d02335cd8370a835f257"
server_key_iv: "825d1e2722c2969dfb690c484728449e"
server_key_version: "encryptionkey@0"
I20260929 16:29:25.578599 16298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:25.602761 16298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:25.606182 16298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:25.606411 16298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:25.613974 16298 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39841
I20260929 16:29:25.614441 16366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39841 every 8 connection(s)
I20260929 16:29:25.615243 16298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:29:25.620137 16367 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:29:25.620455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 16298
I20260929 16:29:25.620874 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46609
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39841
--webserver_port=37137
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:29:25.622038 16367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1: Bootstrap starting.
I20260929 16:29:25.646786 16367 log.cc:824] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:25.658931 16367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:29:25.659404 16367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1: Bootstrap complete.
I20260929 16:29:25.664891 16367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ee0667256042b6834b2b4b596ed0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } }
I20260929 16:29:25.665469 16367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81ee0667256042b6834b2b4b596ed0e1, State: Initialized, Role: FOLLOWER
I20260929 16:29:25.666047 16367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ee0667256042b6834b2b4b596ed0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } }
I20260929 16:29:25.666210 16367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:29:25.666289 16367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:29:25.666390 16367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:29:25.669032 16367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ee0667256042b6834b2b4b596ed0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } }
I20260929 16:29:25.669462 16367 leader_election.cc:302] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81ee0667256042b6834b2b4b596ed0e1; no voters:
I20260929 16:29:25.669835 16367 leader_election.cc:288] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:29:25.670053 16371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:29:25.670569 16371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 2 LEADER]: Becoming Leader. State: Replica: 81ee0667256042b6834b2b4b596ed0e1, State: Running, Role: LEADER
I20260929 16:29:25.671026 16371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ee0667256042b6834b2b4b596ed0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } }
I20260929 16:29:25.671284 16367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:29:25.672703 16373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81ee0667256042b6834b2b4b596ed0e1. Latest consensus state: current_term: 2 leader_uuid: "81ee0667256042b6834b2b4b596ed0e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ee0667256042b6834b2b4b596ed0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } }
I20260929 16:29:25.672860 16373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:25.673303 16372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "81ee0667256042b6834b2b4b596ed0e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ee0667256042b6834b2b4b596ed0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } }
I20260929 16:29:25.673414 16372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:25.677276 16386 catalog_manager.cc:1259] Loaded cluster ID: c1a93fd2a69442d5b83d311a9f4599a9
I20260929 16:29:25.677340 16386 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1: loading cluster ID for follower catalog manager: success
I20260929 16:29:25.679929 16386 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1: acquiring CA information for follower catalog manager: success
I20260929 16:29:25.681557 16386 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:29:25.681742 16382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:29:25.684554 16382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:29:25.685295 16382 catalog_manager.cc:1259] Loaded cluster ID: c1a93fd2a69442d5b83d311a9f4599a9
I20260929 16:29:25.685365 16382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:29:25.686123 16382 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:29:25.686652 16382 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1: Loaded TSK: 0
I20260929 16:29:25.706362 16382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1: Generated new TSK 1
I20260929 16:29:25.706698 16382 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:29:25.764643 16369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:25.764966 16369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:25.765054 16369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:25.769464 16369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:25.769562 16369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:25.769657 16369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:29:25.775625 16369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37137
--tserver_master_addrs=127.0.98.62:39841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:29:25.776778 16369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:25.778028 16369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:25.788560 16395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:25.788559 16394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:25.788895 16397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:25.789242 16369 server_base.cc:1034] running on GCE node
I20260929 16:29:25.789716 16369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:25.790455 16369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:25.791661 16369 hybrid_clock.cc:648] HybridClock initialized: now 1790699365791633 us; error 50 us; skew 500 ppm
Sep 29 16:29:25 dist-test-slave-vq9v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:25.794725 16369 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:29:25.795773 16369 webserver.cc:460] Webserver started at http://127.0.98.1:37137/ using document root <none> and password file <none>
I20260929 16:29:25.796380 16369 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:25.796460 16369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:25.800747 16369 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:29:25.803409 16404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:25.804986 16369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:29:25.805174 16369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "31ed5e5c9cc0450783fa75521dff94e4"
format_stamp: "Formatted at 2026-09-29 16:29:04 on dist-test-slave-vq9v"
server_key: "88138fa4052858362669b96776625c4e"
server_key_iv: "8b35f320572a016105532f2d4d834bc7"
server_key_version: "encryptionkey@0"
I20260929 16:29:25.805919 16369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:25.818266 16369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:25.822093 16369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:25.822331 16369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:25.823037 16369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:25.824256 16411 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 16:29:25.828492 16369 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 16:29:25.828572 16369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260929 16:29:25.828629 16369 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 16:29:25.830482 16369 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 16:29:25.830549 16369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260929 16:29:25.830801 16411 tablet_bootstrap.cc:492] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4: Bootstrap starting.
I20260929 16:29:25.843335 16369 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46609
I20260929 16:29:25.843379 16518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46609 every 8 connection(s)
I20260929 16:29:25.844741 16369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:29:25.849391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 16369
I20260929 16:29:25.849915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37725
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39841
--webserver_port=42221
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:29:25.853006 16411 log.cc:824] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:29:25 dist-test-slave-vq9v krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:29:25.863523 16411 tablet_bootstrap.cc:492] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:29:25.864064 16411 tablet_bootstrap.cc:492] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4: Bootstrap complete.
I20260929 16:29:25.864846 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:25.847217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58345 (local address 127.0.98.62:39841)
0929 16:29:25.847585 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:25.847605 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:25.847968 (+ 363us) server_negotiation.cc:407] Connection header received
0929 16:29:25.849225 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:25.849237 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:25.849667 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:25.850039 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:25.851216 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:25.852864 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:25.854218 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:25.854761 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:25.854959 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:25.858352 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:25.858358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:25.858377 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:25.858415 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:25.860735 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:25.861567 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:25.861573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:25.861580 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:25.861745 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:25.862324 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:25.862328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:25.862330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:25.862813 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:25.862953 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:25.863645 (+ 692us) server_negotiation.cc:299] Negotiation successful
0929 16:29:25.863979 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":134,"threads_started":1}
I20260929 16:29:25.864938 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:25.847198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.1:58345)
0929 16:29:25.847785 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:25.847831 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:25.848745 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:25.850080 (+ 1335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:25.850090 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:25.850553 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:25.851028 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:25.851044 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:25.853043 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:25.853046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:25.854080 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:25.854091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:25.854934 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:25.854942 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:25.855204 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:25.855998 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:25.856026 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:25.858088 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:25.860894 (+ 2806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:25.860905 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:25.860924 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:25.861393 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:25.861859 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:25.861862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:25.861864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:25.862179 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:25.862940 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:25.862947 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:25.863261 (+ 314us) client_negotiation.cc:769] Sending connection context
0929 16:29:25.863456 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 16:29:25.863858 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":161,"threads_started":1}
I20260929 16:29:25.865526 16411 ts_tablet_manager.cc:1397] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4: Time spent bootstrapping tablet: real 0.035s user 0.022s sys 0.011s
I20260929 16:29:25.867940 16519 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39841
I20260929 16:29:25.868184 16519 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:25.868754 16519 heartbeater.cc:511] Master 127.0.98.62:39841 requested a full tablet report, sending...
I20260929 16:29:25.868332 16411 raft_consensus.cc:348] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:25.868852 16411 raft_consensus.cc:729] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31ed5e5c9cc0450783fa75521dff94e4, State: Initialized, Role: FOLLOWER
I20260929 16:29:25.869423 16411 consensus_queue.cc:260] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:25.870568 16411 ts_tablet_manager.cc:1428] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 16:29:25.872452 16331 ts_manager.cc:194] Registered new tserver with Master: 31ed5e5c9cc0450783fa75521dff94e4 (127.0.98.1:46609)
I20260929 16:29:25.877161 16331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58345
I20260929 16:29:25.880362 16519 heartbeater.cc:503] Master 127.0.98.62:39841 was elected leader, sending a full tablet report...
Sep 29 16:29:25 dist-test-slave-vq9v krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 16:29:25.888484 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:25.876498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51944 (local address 127.0.98.1:46609)
0929 16:29:25.877324 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:25.877339 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:25.877365 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:29:25.877445 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:25.877449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:25.877615 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:25.877772 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:25.878542 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:25.879650 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:25.881037 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:25.881522 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:25.881650 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:25.884601 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:25.884610 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:25.884620 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:25.884644 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:25.886410 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:25.887018 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:25.887023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:25.887024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:25.887104 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:25.887399 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:25.887402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:25.887403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:25.887687 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:25.887791 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:25.888203 (+ 412us) server_negotiation.cc:299] Negotiation successful
0929 16:29:25.888347 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":98,"threads_started":1}
I20260929 16:29:25.888545 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:25.876419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46609 (local address 127.0.0.1:51944)
0929 16:29:25.877032 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:25.877080 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:25.877253 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:25.877773 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:25.877779 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:25.878253 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:25.878398 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:25.878406 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:25.879799 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:25.879803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:25.880884 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:25.880908 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:25.881637 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:25.881653 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:25.881831 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:25.882472 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:25.882488 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:25.884450 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:25.886537 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:25.886548 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:25.886554 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:25.886883 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:25.887196 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:25.887200 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:25.887202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:25.887306 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:25.887805 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:25.887811 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:25.887971 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 16:29:25.888132 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 16:29:25.888356 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":160,"threads_started":1}
I20260929 16:29:25.889839 16453 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ab89fca00524db0980f8523ff49635d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51944
I20260929 16:29:25.890975 16528 tablet_replica.cc:330] stopping tablet replica
I20260929 16:29:25.891320 16528 raft_consensus.cc:2229] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:29:25.891580 16528 raft_consensus.cc:2258] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:25.892599 16528 ts_tablet_manager.cc:1905] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:25.896409 16528 ts_tablet_manager.cc:1918] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:29:25.896507 16528 log.cc:1196] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/wal/wals/8ab89fca00524db0980f8523ff49635d
I20260929 16:29:25.896798 16528 ts_tablet_manager.cc:1939] T 8ab89fca00524db0980f8523ff49635d P 31ed5e5c9cc0450783fa75521dff94e4: Deleting consensus metadata
I20260929 16:29:25.898643 16318 catalog_manager.cc:4841] TS 31ed5e5c9cc0450783fa75521dff94e4 (127.0.98.1:46609): tablet 8ab89fca00524db0980f8523ff49635d (table test-table [id=df37e69492664bb5b1a5db817f7efa22]) successfully deleted
W20260929 16:29:25.979291 16523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:25.979607 16523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:25.979684 16523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:25.983946 16523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:25.984041 16523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:25.984133 16523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:29:25.989614 16523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42221
--tserver_master_addrs=127.0.98.62:39841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:29:25.990782 16523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:25.991850 16523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:26.002208 16534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:26.002159 16536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:26.002319 16533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:26.002775 16523 server_base.cc:1034] running on GCE node
I20260929 16:29:26.003198 16523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:26.003856 16523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:26.005077 16523 hybrid_clock.cc:648] HybridClock initialized: now 1790699366005068 us; error 32 us; skew 500 ppm
Sep 29 16:29:26 dist-test-slave-vq9v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:26.008256 16523 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:29:26.009377 16523 webserver.cc:460] Webserver started at http://127.0.98.2:42221/ using document root <none> and password file <none>
I20260929 16:29:26.010046 16523 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:26.010174 16523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:26.015010 16523 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 16:29:26.017531 16543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:26.018972 16523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:29:26.019141 16523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "28c94367cb9e4c69b932928e198b4fb2"
format_stamp: "Formatted at 2026-09-29 16:29:04 on dist-test-slave-vq9v"
server_key: "8ac47c2d2b67ddf272aa5afaeb9aa05d"
server_key_iv: "60957d64c935970ae8ced538997ede1f"
server_key_version: "encryptionkey@0"
I20260929 16:29:26.019768 16523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:26.032872 16523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:26.036513 16523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:26.036741 16523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:26.037395 16523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:26.038272 16523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:26.038347 16523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:26.038429 16523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:26.038475 16523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:26.047694 16523 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37725
I20260929 16:29:26.047792 16656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37725 every 8 connection(s)
I20260929 16:29:26.048761 16523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:29:26.058192 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 16523
I20260929 16:29:26.058753 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37331
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39841
--webserver_port=43989
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:29:26 dist-test-slave-vq9v krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:29:26.066082 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.050857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37431 (local address 127.0.98.62:39841)
0929 16:29:26.051022 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:26.051028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:26.052071 (+ 1043us) server_negotiation.cc:407] Connection header received
0929 16:29:26.053619 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:26.053622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:26.053679 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:26.053767 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:26.055112 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.055830 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.057158 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.057541 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.057640 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.061301 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:26.061305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:26.061307 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:26.061339 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:26.063187 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:26.063984 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:26.063987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:26.063988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:26.064078 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:26.064601 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:26.064603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:26.064604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:26.064807 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:26.064899 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:26.065789 (+ 890us) server_negotiation.cc:299] Negotiation successful
0929 16:29:26.065941 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:29:26.067464 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.051373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.2:37431)
0929 16:29:26.051890 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:26.051936 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:26.053360 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:26.053950 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:26.053958 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:26.054419 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:26.054946 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:26.054960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.055971 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.055974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:26.057029 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:26.057039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.057697 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.057700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:26.057915 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.058718 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:26.058742 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:26.060969 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:26.063336 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:26.063347 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:26.063370 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:26.063837 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:26.064191 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:26.064198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:26.064201 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:26.064482 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:26.064933 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:26.064943 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:26.065414 (+ 471us) client_negotiation.cc:769] Sending connection context
0929 16:29:26.065697 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 16:29:26.066162 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":132,"threads_started":1}
I20260929 16:29:26.069010 16657 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39841
I20260929 16:29:26.069406 16657 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:26.070071 16657 heartbeater.cc:511] Master 127.0.98.62:39841 requested a full tablet report, sending...
I20260929 16:29:26.071431 16331 ts_manager.cc:194] Registered new tserver with Master: 28c94367cb9e4c69b932928e198b4fb2 (127.0.98.2:37725)
I20260929 16:29:26.072280 16331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37431
W20260929 16:29:26.176589 16660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:26.176857 16660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:26.176913 16660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:26.181063 16660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:26.181140 16660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:26.181288 16660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:29:26.187070 16660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43989
--tserver_master_addrs=127.0.98.62:39841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:29:26.188191 16660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:26.189328 16660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:26.199528 16666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:26.199558 16668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:26.199558 16665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:26.200024 16660 server_base.cc:1034] running on GCE node
I20260929 16:29:26.200490 16660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:26.201138 16660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:26.202296 16660 hybrid_clock.cc:648] HybridClock initialized: now 1790699366202262 us; error 48 us; skew 500 ppm
Sep 29 16:29:26 dist-test-slave-vq9v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:26.205565 16660 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:29:26.206815 16660 webserver.cc:460] Webserver started at http://127.0.98.3:43989/ using document root <none> and password file <none>
I20260929 16:29:26.207605 16660 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:26.207721 16660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:26.212663 16660 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:29:26.215031 16675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:26.216430 16660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:29:26.216571 16660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "2e1c0eed17954a1cac7908f34e3d8b63"
format_stamp: "Formatted at 2026-09-29 16:29:05 on dist-test-slave-vq9v"
server_key: "b90ac823f93da59a388fecc6ff796269"
server_key_iv: "648534cbe3d0968e319710ba4ee21f0f"
server_key_version: "encryptionkey@0"
I20260929 16:29:26.217155 16660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:26.231222 16660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:26.234530 16660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:26.234746 16660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:26.235374 16660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:26.236251 16660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:26.236297 16660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:26.236337 16660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:26.236353 16660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:26.245589 16660 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37331
I20260929 16:29:26.245627 16788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37331 every 8 connection(s)
I20260929 16:29:26.246631 16660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 16:29:26 dist-test-slave-vq9v krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:29:26.256564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 16660
I20260929 16:29:26.261183 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.248539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56065 (local address 127.0.98.62:39841)
0929 16:29:26.248695 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:26.248699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:26.249598 (+ 899us) server_negotiation.cc:407] Connection header received
0929 16:29:26.250926 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:26.250929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:26.250987 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:26.251075 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:26.252238 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.252766 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.253955 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.254199 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.254316 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.256863 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:26.256866 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:26.256869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:26.256894 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:26.258775 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:26.259440 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:26.259444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:26.259446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:26.259531 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:26.259959 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:26.259962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:26.259963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:26.260149 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:26.260277 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:26.260835 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 16:29:26.260987 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:29:26.262003 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.248959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.3:56065)
0929 16:29:26.249445 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:26.249488 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:26.250709 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:26.251259 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:26.251266 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:26.251624 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:26.252086 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:26.252098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.252908 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.252914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:26.253826 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:26.253835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.254361 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.254365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:26.254533 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.255106 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:26.255122 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:26.256671 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:26.258913 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:26.258920 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:26.258941 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:26.259313 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:26.259636 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:26.259639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:26.259642 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:26.259851 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:26.260275 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:26.260283 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:26.260552 (+ 269us) client_negotiation.cc:769] Sending connection context
0929 16:29:26.260781 (+ 229us) client_negotiation.cc:240] Negotiation successful
0929 16:29:26.261130 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":101,"threads_started":1}
I20260929 16:29:26.263554 16789 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39841
I20260929 16:29:26.263804 16789 heartbeater.cc:464] Registering TS with master...
I20260929 16:29:26.264325 16789 heartbeater.cc:511] Master 127.0.98.62:39841 requested a full tablet report, sending...
I20260929 16:29:26.265615 16331 ts_manager.cc:194] Registered new tserver with Master: 2e1c0eed17954a1cac7908f34e3d8b63 (127.0.98.3:37331)
I20260929 16:29:26.266225 16331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56065
I20260929 16:29:26.266875 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.257330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57806 (local address 127.0.98.62:39841)
0929 16:29:26.257603 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:26.257609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:26.258107 (+ 498us) server_negotiation.cc:407] Connection header received
0929 16:29:26.258195 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:26.258200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:26.258271 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:26.258432 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:26.259255 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.260086 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.261081 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.261615 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.261750 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.263391 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:26.263395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:26.263398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:26.263425 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:26.265168 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:26.265632 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:26.265637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:26.265639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:26.265720 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:26.266009 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:26.266013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:26.266015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:26.266185 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:26.266406 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:26.266562 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 16:29:26.266682 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":84,"threads_started":1}
I20260929 16:29:26.268893 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:29:26.273476 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.269531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.0.98.62:39841)
0929 16:29:26.269654 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:26.269657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:26.269898 (+ 241us) server_negotiation.cc:407] Connection header received
0929 16:29:26.269983 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:26.269985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:26.270067 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:26.270151 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:29:26.270415 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.271112 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.272071 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.272318 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.272425 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.272743 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:29:26.272870 (+ 127us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 16:29:26.273337 (+ 467us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57818: token expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 16:29:26.273545 16793 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57818: token expired
W20260929 16:29:26.273885 16260 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:39841)
I20260929 16:29:26.282606 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.274318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57830 (local address 127.0.98.62:39841)
0929 16:29:26.274441 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:26.274447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:26.274464 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:29:26.274576 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:26.274581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:26.274635 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:26.274727 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:26.275357 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.276167 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.276979 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.277254 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.277367 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.279280 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:26.279284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:26.279287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:26.279325 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:26.280775 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:26.281249 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:26.281253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:26.281256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:26.281346 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:26.281723 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:26.281726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:26.281728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:26.281979 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:26.282107 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:26.282352 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 16:29:26.282479 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:29:26.285606 16331 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:29:26.289388 16331 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:29:26.298101 16453 tablet_service.cc:1467] Processing CreateTablet for tablet 814c316fbda84c7a859b92d60fb97645 (DEFAULT_TABLE table=test-table [id=e7a79d7726c94791ad628f7e74dac30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:26.298842 16453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 814c316fbda84c7a859b92d60fb97645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:26.301947 16411 tablet_bootstrap.cc:492] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4: Bootstrap starting.
I20260929 16:29:26.302588 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.296841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37331 (local address 127.0.0.1:59282)
0929 16:29:26.297012 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:26.297040 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:26.297139 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:26.298143 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:26.298149 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:26.298175 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:26.298339 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:26.298348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.299959 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.299963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:26.300963 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:26.300973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.302207 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.302212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:26.302320 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.302363 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 16:29:26.302414 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 16:29:26.302480 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 16:29:26.302922 16411 tablet_bootstrap.cc:654] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:26.303268 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.297047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59282 (local address 127.0.98.3:37331)
0929 16:29:26.297560 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:26.297575 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:26.297602 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:29:26.297693 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:26.297699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:26.297957 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:26.298154 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:26.298527 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.299824 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.301110 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.302053 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.302814 (+ 761us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.302998 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:26.303061 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 16:29:26.303153 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":158,"threads_started":1}
I20260929 16:29:26.303826 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.297185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37725 (local address 127.0.0.1:57734)
0929 16:29:26.298112 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:26.298127 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:26.298219 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:26.299024 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:26.299027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:26.299047 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:26.299160 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:26.299169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.300454 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.300462 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:26.301783 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:26.301795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.303332 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.303339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:26.303465 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.303498 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:29:26.303586 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 16:29:26.303682 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":1,"thread_start_us":227,"threads_started":1}
I20260929 16:29:26.304101 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.297667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57734 (local address 127.0.98.2:37725)
0929 16:29:26.298012 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:26.298026 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:26.298334 (+ 308us) server_negotiation.cc:407] Connection header received
0929 16:29:26.298422 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:26.298430 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:26.298643 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:26.298805 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:26.299314 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.300307 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.301943 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.302938 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.303057 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.303228 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:26.303640 (+ 412us) server_negotiation.cc:299] Negotiation successful
0929 16:29:26.303748 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":158,"threads_started":1}
I20260929 16:29:26.304629 16411 tablet_bootstrap.cc:492] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4: No bootstrap required, opened a new log
I20260929 16:29:26.304733 16411 ts_tablet_manager.cc:1397] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 16:29:26.305249 16411 raft_consensus.cc:348] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:26.305380 16411 raft_consensus.cc:374] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:26.305423 16411 raft_consensus.cc:729] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31ed5e5c9cc0450783fa75521dff94e4, State: Initialized, Role: FOLLOWER
I20260929 16:29:26.305558 16411 consensus_queue.cc:260] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:26.305833 16411 ts_tablet_manager.cc:1428] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 16:29:26.307229 16591 tablet_service.cc:1467] Processing CreateTablet for tablet 814c316fbda84c7a859b92d60fb97645 (DEFAULT_TABLE table=test-table [id=e7a79d7726c94791ad628f7e74dac30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:26.307236 16723 tablet_service.cc:1467] Processing CreateTablet for tablet 814c316fbda84c7a859b92d60fb97645 (DEFAULT_TABLE table=test-table [id=e7a79d7726c94791ad628f7e74dac30f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:29:26.308414 16591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 814c316fbda84c7a859b92d60fb97645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:26.308420 16723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 814c316fbda84c7a859b92d60fb97645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:29:26.315975 16802 tablet_bootstrap.cc:492] T 814c316fbda84c7a859b92d60fb97645 P 28c94367cb9e4c69b932928e198b4fb2: Bootstrap starting.
I20260929 16:29:26.316459 16803 tablet_bootstrap.cc:492] T 814c316fbda84c7a859b92d60fb97645 P 2e1c0eed17954a1cac7908f34e3d8b63: Bootstrap starting.
I20260929 16:29:26.318643 16803 tablet_bootstrap.cc:654] T 814c316fbda84c7a859b92d60fb97645 P 2e1c0eed17954a1cac7908f34e3d8b63: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:26.319046 16802 tablet_bootstrap.cc:654] T 814c316fbda84c7a859b92d60fb97645 P 28c94367cb9e4c69b932928e198b4fb2: Neither blocks nor log segments found. Creating new log.
I20260929 16:29:26.319826 16802 log.cc:824] T 814c316fbda84c7a859b92d60fb97645 P 28c94367cb9e4c69b932928e198b4fb2: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:26.319826 16803 log.cc:824] T 814c316fbda84c7a859b92d60fb97645 P 2e1c0eed17954a1cac7908f34e3d8b63: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:26.322280 16803 tablet_bootstrap.cc:492] T 814c316fbda84c7a859b92d60fb97645 P 2e1c0eed17954a1cac7908f34e3d8b63: No bootstrap required, opened a new log
I20260929 16:29:26.322290 16802 tablet_bootstrap.cc:492] T 814c316fbda84c7a859b92d60fb97645 P 28c94367cb9e4c69b932928e198b4fb2: No bootstrap required, opened a new log
I20260929 16:29:26.322516 16803 ts_tablet_manager.cc:1397] T 814c316fbda84c7a859b92d60fb97645 P 2e1c0eed17954a1cac7908f34e3d8b63: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:29:26.322521 16802 ts_tablet_manager.cc:1397] T 814c316fbda84c7a859b92d60fb97645 P 28c94367cb9e4c69b932928e198b4fb2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 16:29:26.326049 16803 raft_consensus.cc:348] T 814c316fbda84c7a859b92d60fb97645 P 2e1c0eed17954a1cac7908f34e3d8b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:26.326050 16802 raft_consensus.cc:348] T 814c316fbda84c7a859b92d60fb97645 P 28c94367cb9e4c69b932928e198b4fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:26.326320 16802 raft_consensus.cc:374] T 814c316fbda84c7a859b92d60fb97645 P 28c94367cb9e4c69b932928e198b4fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:26.326347 16803 raft_consensus.cc:374] T 814c316fbda84c7a859b92d60fb97645 P 2e1c0eed17954a1cac7908f34e3d8b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:29:26.326361 16802 raft_consensus.cc:729] T 814c316fbda84c7a859b92d60fb97645 P 28c94367cb9e4c69b932928e198b4fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28c94367cb9e4c69b932928e198b4fb2, State: Initialized, Role: FOLLOWER
I20260929 16:29:26.326417 16803 raft_consensus.cc:729] T 814c316fbda84c7a859b92d60fb97645 P 2e1c0eed17954a1cac7908f34e3d8b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1c0eed17954a1cac7908f34e3d8b63, State: Initialized, Role: FOLLOWER
I20260929 16:29:26.326870 16802 consensus_queue.cc:260] T 814c316fbda84c7a859b92d60fb97645 P 28c94367cb9e4c69b932928e198b4fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:26.326887 16803 consensus_queue.cc:260] T 814c316fbda84c7a859b92d60fb97645 P 2e1c0eed17954a1cac7908f34e3d8b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:26.327665 16657 heartbeater.cc:503] Master 127.0.98.62:39841 was elected leader, sending a full tablet report...
I20260929 16:29:26.327878 16789 heartbeater.cc:503] Master 127.0.98.62:39841 was elected leader, sending a full tablet report...
I20260929 16:29:26.327996 16802 ts_tablet_manager.cc:1428] T 814c316fbda84c7a859b92d60fb97645 P 28c94367cb9e4c69b932928e198b4fb2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 16:29:26.327996 16803 ts_tablet_manager.cc:1428] T 814c316fbda84c7a859b92d60fb97645 P 2e1c0eed17954a1cac7908f34e3d8b63: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260929 16:29:26.346251 16520 tablet.cc:2369] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:29:26.382858 16525 raft_consensus.cc:482] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:29:26.383142 16525 raft_consensus.cc:504] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:26.384385 16525 leader_election.cc:288] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e1c0eed17954a1cac7908f34e3d8b63 (127.0.98.3:37331), 28c94367cb9e4c69b932928e198b4fb2 (127.0.98.2:37725)
I20260929 16:29:26.388567 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.384985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36227 (local address 127.0.98.3:37331)
0929 16:29:26.385108 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:26.385112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:26.385315 (+ 203us) server_negotiation.cc:407] Connection header received
0929 16:29:26.385541 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:26.385544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:26.385601 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:26.385687 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:26.385952 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.386461 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.387542 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.387969 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.388093 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.388154 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:26.388355 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 16:29:26.388428 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:29:26.388586 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.384753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37331 (local address 127.0.98.1:36227)
0929 16:29:26.385190 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:26.385208 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:26.385379 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:26.385701 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:26.385703 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:26.385720 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:26.385841 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:26.385847 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.386578 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.386581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:26.387409 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:26.387426 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.388099 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.388103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:26.388198 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.388249 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 16:29:26.388339 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 16:29:26.388437 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":1,"thread_start_us":105,"threads_started":1}
I20260929 16:29:26.388572 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.384884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48855 (local address 127.0.98.2:37725)
0929 16:29:26.385026 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:26.385031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:26.385486 (+ 455us) server_negotiation.cc:407] Connection header received
0929 16:29:26.385601 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:26.385604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:26.385657 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:26.385743 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:29:26.386052 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.386561 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.387579 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.387990 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.388116 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.388179 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:26.388373 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 16:29:26.388448 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:29:26.388604 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.384753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37725 (local address 127.0.98.1:48855)
0929 16:29:26.385359 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:26.385372 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:26.385499 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:26.385766 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:26.385769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:26.385785 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:29:26.385933 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:26.385939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.386678 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.386683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:26.387444 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:26.387452 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.388135 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.388139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:26.388237 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.388258 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:29:26.388356 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 16:29:26.388436 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":60,"thread_start_us":91,"threads_started":1}
I20260929 16:29:26.389348 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "814c316fbda84c7a859b92d60fb97645" candidate_uuid: "31ed5e5c9cc0450783fa75521dff94e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" is_pre_election: true
I20260929 16:29:26.389479 16611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "814c316fbda84c7a859b92d60fb97645" candidate_uuid: "31ed5e5c9cc0450783fa75521dff94e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28c94367cb9e4c69b932928e198b4fb2" is_pre_election: true
I20260929 16:29:26.389686 16743 raft_consensus.cc:2454] T 814c316fbda84c7a859b92d60fb97645 P 2e1c0eed17954a1cac7908f34e3d8b63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31ed5e5c9cc0450783fa75521dff94e4 in term 0.
I20260929 16:29:26.389756 16611 raft_consensus.cc:2454] T 814c316fbda84c7a859b92d60fb97645 P 28c94367cb9e4c69b932928e198b4fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31ed5e5c9cc0450783fa75521dff94e4 in term 0.
I20260929 16:29:26.390246 16406 leader_election.cc:302] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28c94367cb9e4c69b932928e198b4fb2, 2e1c0eed17954a1cac7908f34e3d8b63, 31ed5e5c9cc0450783fa75521dff94e4; no voters:
I20260929 16:29:26.390547 16525 raft_consensus.cc:2781] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:29:26.390635 16525 raft_consensus.cc:482] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:29:26.390678 16525 raft_consensus.cc:3037] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:26.391673 16525 raft_consensus.cc:504] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:26.392056 16525 leader_election.cc:288] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 1 election: Requested vote from peers 2e1c0eed17954a1cac7908f34e3d8b63 (127.0.98.3:37331), 28c94367cb9e4c69b932928e198b4fb2 (127.0.98.2:37725)
I20260929 16:29:26.392357 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "814c316fbda84c7a859b92d60fb97645" candidate_uuid: "31ed5e5c9cc0450783fa75521dff94e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1c0eed17954a1cac7908f34e3d8b63"
I20260929 16:29:26.392514 16743 raft_consensus.cc:3037] T 814c316fbda84c7a859b92d60fb97645 P 2e1c0eed17954a1cac7908f34e3d8b63 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:26.392554 16611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "814c316fbda84c7a859b92d60fb97645" candidate_uuid: "31ed5e5c9cc0450783fa75521dff94e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28c94367cb9e4c69b932928e198b4fb2"
I20260929 16:29:26.392673 16611 raft_consensus.cc:3037] T 814c316fbda84c7a859b92d60fb97645 P 28c94367cb9e4c69b932928e198b4fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:29:26.393968 16743 raft_consensus.cc:2454] T 814c316fbda84c7a859b92d60fb97645 P 2e1c0eed17954a1cac7908f34e3d8b63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ed5e5c9cc0450783fa75521dff94e4 in term 1.
I20260929 16:29:26.393965 16611 raft_consensus.cc:2454] T 814c316fbda84c7a859b92d60fb97645 P 28c94367cb9e4c69b932928e198b4fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31ed5e5c9cc0450783fa75521dff94e4 in term 1.
I20260929 16:29:26.394357 16406 leader_election.cc:302] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e1c0eed17954a1cac7908f34e3d8b63, 31ed5e5c9cc0450783fa75521dff94e4; no voters:
I20260929 16:29:26.394600 16525 raft_consensus.cc:2781] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:29:26.394822 16525 raft_consensus.cc:686] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 LEADER]: Becoming Leader. State: Replica: 31ed5e5c9cc0450783fa75521dff94e4, State: Running, Role: LEADER
I20260929 16:29:26.395121 16525 consensus_queue.cc:237] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:26.398319 16330 catalog_manager.cc:5495] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31ed5e5c9cc0450783fa75521dff94e4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "31ed5e5c9cc0450783fa75521dff94e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } health_report { overall_health: HEALTHY } } }
I20260929 16:29:26.431249 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.427447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46848 (local address 127.0.98.1:46609)
0929 16:29:26.427711 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:26.427715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:26.427727 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:29:26.427804 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:26.427806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:26.427858 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:26.427961 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:29:26.428246 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.428777 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.429743 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:26.429970 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:26.430048 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:26.430299 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:29:26.430749 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:29:26.430839 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:26.431063 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 16:29:26.431117 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":120,"threads_started":1}
I20260929 16:29:26.435407 16743 raft_consensus.cc:1261] T 814c316fbda84c7a859b92d60fb97645 P 2e1c0eed17954a1cac7908f34e3d8b63 [term 1 FOLLOWER]: Refusing update from remote peer 31ed5e5c9cc0450783fa75521dff94e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:29:26.435407 16611 raft_consensus.cc:1261] T 814c316fbda84c7a859b92d60fb97645 P 28c94367cb9e4c69b932928e198b4fb2 [term 1 FOLLOWER]: Refusing update from remote peer 31ed5e5c9cc0450783fa75521dff94e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:29:26.436199 16525 consensus_queue.cc:1035] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:29:26.436444 16810 consensus_queue.cc:1035] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:29:26.444761 16816 mvcc.cc:205] Tried to move back new op lower bound from 7334704604912205824 to 7334704604757229568. Current Snapshot: MvccSnapshot[applied={T|T < 7334704604912205824}]
I20260929 16:29:26.445408 16815 mvcc.cc:205] Tried to move back new op lower bound from 7334704604912205824 to 7334704604757229568. Current Snapshot: MvccSnapshot[applied={T|T < 7334704604912205824}]
I20260929 16:29:26.453019 16331 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:29:26.453280 16331 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:29:26.456082 16331 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1: Sending DeleteTablet for 3 replicas of tablet 814c316fbda84c7a859b92d60fb97645
I20260929 16:29:26.456990 16723 tablet_service.cc:1514] Processing DeleteTablet for tablet 814c316fbda84c7a859b92d60fb97645 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59282
I20260929 16:29:26.457263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 16369
I20260929 16:29:26.457310 16591 tablet_service.cc:1514] Processing DeleteTablet for tablet 814c316fbda84c7a859b92d60fb97645 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:29:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57734
I20260929 16:29:26.458148 16824 tablet_replica.cc:330] T 814c316fbda84c7a859b92d60fb97645 P 2e1c0eed17954a1cac7908f34e3d8b63: stopping tablet replica
I20260929 16:29:26.458541 16824 raft_consensus.cc:2229] T 814c316fbda84c7a859b92d60fb97645 P 2e1c0eed17954a1cac7908f34e3d8b63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:29:26.458923 16824 raft_consensus.cc:2258] T 814c316fbda84c7a859b92d60fb97645 P 2e1c0eed17954a1cac7908f34e3d8b63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:26.459937 16825 tablet_replica.cc:330] T 814c316fbda84c7a859b92d60fb97645 P 28c94367cb9e4c69b932928e198b4fb2: stopping tablet replica
I20260929 16:29:26.460343 16825 raft_consensus.cc:2229] T 814c316fbda84c7a859b92d60fb97645 P 28c94367cb9e4c69b932928e198b4fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:29:26.460654 16825 raft_consensus.cc:2258] T 814c316fbda84c7a859b92d60fb97645 P 28c94367cb9e4c69b932928e198b4fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:29:26.460822 16824 ts_tablet_manager.cc:1905] T 814c316fbda84c7a859b92d60fb97645 P 2e1c0eed17954a1cac7908f34e3d8b63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:26.462002 16825 ts_tablet_manager.cc:1905] T 814c316fbda84c7a859b92d60fb97645 P 28c94367cb9e4c69b932928e198b4fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:29:26.464282 16825 ts_tablet_manager.cc:1918] T 814c316fbda84c7a859b92d60fb97645 P 28c94367cb9e4c69b932928e198b4fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:29:26.464396 16825 log.cc:1196] T 814c316fbda84c7a859b92d60fb97645 P 28c94367cb9e4c69b932928e198b4fb2: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/814c316fbda84c7a859b92d60fb97645
I20260929 16:29:26.464747 16825 ts_tablet_manager.cc:1939] T 814c316fbda84c7a859b92d60fb97645 P 28c94367cb9e4c69b932928e198b4fb2: Deleting consensus metadata
I20260929 16:29:26.464860 16824 ts_tablet_manager.cc:1918] T 814c316fbda84c7a859b92d60fb97645 P 2e1c0eed17954a1cac7908f34e3d8b63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:29:26.464979 16824 log.cc:1196] T 814c316fbda84c7a859b92d60fb97645 P 2e1c0eed17954a1cac7908f34e3d8b63: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/814c316fbda84c7a859b92d60fb97645
I20260929 16:29:26.465415 16824 ts_tablet_manager.cc:1939] T 814c316fbda84c7a859b92d60fb97645 P 2e1c0eed17954a1cac7908f34e3d8b63: Deleting consensus metadata
W20260929 16:29:26.466089 16318 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46609 (error 108)
I20260929 16:29:26.466141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 16523
I20260929 16:29:26.466198 16317 catalog_manager.cc:4841] TS 28c94367cb9e4c69b932928e198b4fb2 (127.0.98.2:37725): tablet 814c316fbda84c7a859b92d60fb97645 (table test-table [id=e7a79d7726c94791ad628f7e74dac30f]) successfully deleted
I20260929 16:29:26.467739 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:26.467348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46609 (local address 127.0.0.1:46854)
0929 16:29:26.467562 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:26.467634 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260929 16:29:26.467913 16318 catalog_manager.cc:4584] TS 31ed5e5c9cc0450783fa75521dff94e4 (127.0.98.1:46609): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 814c316fbda84c7a859b92d60fb97645: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46609: connect: Connection refused (error 111)
I20260929 16:29:26.468791 16316 catalog_manager.cc:4841] TS 2e1c0eed17954a1cac7908f34e3d8b63 (127.0.98.3:37331): tablet 814c316fbda84c7a859b92d60fb97645 (table test-table [id=e7a79d7726c94791ad628f7e74dac30f]) successfully deleted
I20260929 16:29:26.472775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 16660
I20260929 16:29:26.481736 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 16298
I20260929 16:29:46.489053 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39841
--webserver_interface=127.0.98.62
--webserver_port=42755
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:46.602927 16827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:46.603221 16827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:46.603264 16827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:46.607487 16827 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:29:46.607564 16827 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:29:46.607583 16827 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:29:46.607597 16827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:29:46.607616 16827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:46.607635 16827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:46.607678 16827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:29:46.607715 16827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:29:46.613251 16827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:29:46.614459 16827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:46.615578 16827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:46.624572 16832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:46.624589 16833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:46.624572 16835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:46.625020 16827 server_base.cc:1034] running on GCE node
I20260929 16:29:46.625675 16827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:46.626967 16827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:46.628180 16827 hybrid_clock.cc:648] HybridClock initialized: now 1790699386628165 us; error 32 us; skew 500 ppm
Sep 29 16:29:46 dist-test-slave-vq9v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:46.635627 16827 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:29:46.636778 16827 webserver.cc:460] Webserver started at http://127.0.98.62:42755/ using document root <none> and password file <none>
I20260929 16:29:46.637478 16827 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:46.637580 16827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:46.641790 16827 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:29:46.644124 16842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:46.645545 16827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:29:46.645699 16827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "81ee0667256042b6834b2b4b596ed0e1"
format_stamp: "Formatted at 2026-09-29 16:29:04 on dist-test-slave-vq9v"
server_key: "d2c058f5a5b5d02335cd8370a835f257"
server_key_iv: "825d1e2722c2969dfb690c484728449e"
server_key_version: "encryptionkey@0"
I20260929 16:29:46.646302 16827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:46.672658 16827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:46.676316 16827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:46.676537 16827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:46.683929 16827 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39841
I20260929 16:29:46.684264 16894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39841 every 8 connection(s)
I20260929 16:29:46.685086 16827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:29:46.685954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 16827
I20260929 16:29:46.686403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46609
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39841
--webserver_port=37137
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:29:46.689721 16895 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:29:46.691406 16895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1: Bootstrap starting.
I20260929 16:29:46.708932 16895 log.cc:824] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:46.720578 16895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:29:46.721011 16895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1: Bootstrap complete.
I20260929 16:29:46.725311 16895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ee0667256042b6834b2b4b596ed0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } }
I20260929 16:29:46.725801 16895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81ee0667256042b6834b2b4b596ed0e1, State: Initialized, Role: FOLLOWER
I20260929 16:29:46.726330 16895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ee0667256042b6834b2b4b596ed0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } }
I20260929 16:29:46.726491 16895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:29:46.726588 16895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:29:46.726730 16895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 2 FOLLOWER]: Advancing to term 3
I20260929 16:29:46.728848 16895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ee0667256042b6834b2b4b596ed0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } }
I20260929 16:29:46.729295 16895 leader_election.cc:302] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81ee0667256042b6834b2b4b596ed0e1; no voters:
I20260929 16:29:46.729655 16895 leader_election.cc:288] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 16:29:46.729836 16899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 16:29:46.730229 16899 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [term 3 LEADER]: Becoming Leader. State: Replica: 81ee0667256042b6834b2b4b596ed0e1, State: Running, Role: LEADER
I20260929 16:29:46.730850 16899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ee0667256042b6834b2b4b596ed0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } }
I20260929 16:29:46.730957 16895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:29:46.732420 16901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81ee0667256042b6834b2b4b596ed0e1. Latest consensus state: current_term: 3 leader_uuid: "81ee0667256042b6834b2b4b596ed0e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ee0667256042b6834b2b4b596ed0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } }
I20260929 16:29:46.732556 16901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:46.732880 16900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "81ee0667256042b6834b2b4b596ed0e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ee0667256042b6834b2b4b596ed0e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39841 } } }
I20260929 16:29:46.732969 16900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:29:46.736276 16914 catalog_manager.cc:1259] Loaded cluster ID: c1a93fd2a69442d5b83d311a9f4599a9
I20260929 16:29:46.736390 16914 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1: loading cluster ID for follower catalog manager: success
I20260929 16:29:46.738417 16914 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 81ee0667256042b6834b2b4b596ed0e1: acquiring CA information for follower catalog manager: success
F20260929 16:29:46.739506 16914 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb5b7e9a9ad google::LogMessage::Fail() at ??:0
@ 0x7fb5b7e9d88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb5b7e9a4de google::LogMessage::Flush() at ??:0
@ 0x7fb5b7e9ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb5bab5e18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb5c27ff989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fb5c27dc023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fb5c27dc54b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fb5c27d5652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fb5c27d49e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7fb5c281b541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb5ba0dd512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb5bb7216db start_thread at ??:0
@ 0x7fb5b699861f clone at ??:0
W20260929 16:29:46.825467 16897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:46.825843 16897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:46.825923 16897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:46.830152 16897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:46.830245 16897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:46.830338 16897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:29:46.836115 16897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37137
--tserver_master_addrs=127.0.98.62:39841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:29:46.837402 16897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:46.838479 16897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:46.848618 16938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:46.848733 16935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:46.848840 16936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:46.848901 16897 server_base.cc:1034] running on GCE node
I20260929 16:29:46.849392 16897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:46.850123 16897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:46.851351 16897 hybrid_clock.cc:648] HybridClock initialized: now 1790699386851329 us; error 42 us; skew 500 ppm
Sep 29 16:29:46 dist-test-slave-vq9v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:46.854754 16897 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:29:46.855786 16897 webserver.cc:460] Webserver started at http://127.0.98.1:37137/ using document root <none> and password file <none>
I20260929 16:29:46.856398 16897 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:46.856462 16897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:46.860890 16897 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:29:46.863494 16945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:46.864894 16897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:29:46.865039 16897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "31ed5e5c9cc0450783fa75521dff94e4"
format_stamp: "Formatted at 2026-09-29 16:29:04 on dist-test-slave-vq9v"
server_key: "88138fa4052858362669b96776625c4e"
server_key_iv: "8b35f320572a016105532f2d4d834bc7"
server_key_version: "encryptionkey@0"
I20260929 16:29:46.865693 16897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:46.882827 16897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:46.886421 16897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:46.886677 16897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:46.887326 16897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:46.888470 16952 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 16:29:46.893957 16897 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 16:29:46.894059 16897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260929 16:29:46.894151 16897 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 16:29:46.895994 16897 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 16:29:46.896080 16897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 16:29:46.896400 16952 tablet_bootstrap.cc:492] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4: Bootstrap starting.
I20260929 16:29:46.907187 16897 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46609
I20260929 16:29:46.907439 17059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46609 every 8 connection(s)
I20260929 16:29:46.908244 16897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:29:46.913523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 16897
I20260929 16:29:46.913959 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37725
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39841
--webserver_port=42221
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:29:46.919363 16952 log.cc:824] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:29:46.929090 16952 tablet_bootstrap.cc:492] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:29:46.929601 16952 tablet_bootstrap.cc:492] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4: Bootstrap complete.
I20260929 16:29:46.931093 16952 ts_tablet_manager.cc:1397] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4: Time spent bootstrapping tablet: real 0.035s user 0.024s sys 0.008s
I20260929 16:29:46.934577 16952 raft_consensus.cc:348] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:46.935153 16952 raft_consensus.cc:729] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31ed5e5c9cc0450783fa75521dff94e4, State: Initialized, Role: FOLLOWER
I20260929 16:29:46.935752 16952 consensus_queue.cc:260] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:46.936899 16952 ts_tablet_manager.cc:1428] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260929 16:29:46.971455 17062 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:29:46.910281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.1:52145)
0929 16:29:46.910777 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:46.910832 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:46.912037 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:46.970627 (+ 58590us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":139,"threads_started":1}
W20260929 16:29:46.972167 17060 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39841 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39841: Client connection negotiation failed: client connection to 127.0.98.62:39841: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 16:29:46.972973 17062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:46.972685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.1:46309)
0929 16:29:46.972824 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:46.972907 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 16:29:46.973829 17062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:46.973535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.1:60515)
0929 16:29:46.973710 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:46.973769 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260929 16:29:46.974130 17060 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:29:47.034467 17063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:47.034826 17063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:47.034878 17063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:47.039269 17063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:47.039353 17063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:47.039415 17063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:29:47.045114 17063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42221
--tserver_master_addrs=127.0.98.62:39841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:29:47.046306 17063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:47.047456 17063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:29:47.057479 17073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:47.057499 17070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:47.057479 17071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:47.058058 17063 server_base.cc:1034] running on GCE node
I20260929 16:29:47.058475 17063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:47.059151 17063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:47.060316 17063 hybrid_clock.cc:648] HybridClock initialized: now 1790699387060288 us; error 48 us; skew 500 ppm
Sep 29 16:29:47 dist-test-slave-vq9v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:47.063237 17063 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:29:47.064278 17063 webserver.cc:460] Webserver started at http://127.0.98.2:42221/ using document root <none> and password file <none>
I20260929 16:29:47.064877 17063 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:47.064939 17063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:47.069301 17063 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:29:47.071748 17080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:47.073149 17063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:29:47.073354 17063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "28c94367cb9e4c69b932928e198b4fb2"
format_stamp: "Formatted at 2026-09-29 16:29:04 on dist-test-slave-vq9v"
server_key: "8ac47c2d2b67ddf272aa5afaeb9aa05d"
server_key_iv: "60957d64c935970ae8ced538997ede1f"
server_key_version: "encryptionkey@0"
I20260929 16:29:47.073946 17063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:47.104014 17063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:47.107300 17063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:47.107517 17063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:47.108150 17063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:47.109009 17063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:47.109081 17063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:47.109151 17063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:47.109194 17063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:47.122982 17063 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37725
I20260929 16:29:47.123014 17193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37725 every 8 connection(s)
I20260929 16:29:47.124195 17063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:29:47.128679 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:47.126715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.2:57437)
0929 16:29:47.127248 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:47.127827 (+ 579us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":116,"threads_started":1}
W20260929 16:29:47.129266 17194 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39841 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39841: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
I20260929 16:29:47.129947 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:47.129689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.2:40075)
0929 16:29:47.129826 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:47.129884 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 16:29:47.130642 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:47.130410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.2:43689)
0929 16:29:47.130533 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:47.130582 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 16:29:47.130859 17194 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:29:47.132587 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 17063
I20260929 16:29:47.133044 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37331
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39841
--webserver_port=43989
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:29:47.245710 17197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:29:47.245990 17197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:29:47.246037 17197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:29:47.250100 17197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:29:47.250204 17197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:29:47.250317 17197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:29:47.255786 17197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43989
--tserver_master_addrs=127.0.98.62:39841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:29:47.256919 17197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:29:47.258064 17197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:29:47.268937 17197 server_base.cc:1034] running on GCE node
W20260929 16:29:47.268895 17203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:47.268895 17202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:29:47.268929 17205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:29:47.269663 17197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:29:47.270481 17197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:29:47.271667 17197 hybrid_clock.cc:648] HybridClock initialized: now 1790699387271651 us; error 29 us; skew 500 ppm
Sep 29 16:29:47 dist-test-slave-vq9v krb5kdc[15751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:29:47.274551 17197 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:29:47.275460 17197 webserver.cc:460] Webserver started at http://127.0.98.3:43989/ using document root <none> and password file <none>
I20260929 16:29:47.276041 17197 fs_manager.cc:359] Metadata directory not provided
I20260929 16:29:47.276134 17197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:29:47.280385 17197 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:29:47.283138 17212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:47.284539 17197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:29:47.284674 17197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "2e1c0eed17954a1cac7908f34e3d8b63"
format_stamp: "Formatted at 2026-09-29 16:29:05 on dist-test-slave-vq9v"
server_key: "b90ac823f93da59a388fecc6ff796269"
server_key_iv: "648534cbe3d0968e319710ba4ee21f0f"
server_key_version: "encryptionkey@0"
I20260929 16:29:47.285326 17197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:29:47.298586 17197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:29:47.301959 17197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:29:47.302184 17197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:29:47.302809 17197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:29:47.303655 17197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:29:47.303735 17197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:47.303812 17197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:29:47.303864 17197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:29:47.313545 17197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37331
I20260929 16:29:47.313619 17325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37331 every 8 connection(s)
I20260929 16:29:47.314671 17197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:29:47.319130 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:47.317096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.3:54031)
0929 16:29:47.317698 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:47.318203 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":125,"threads_started":1}
W20260929 16:29:47.319623 17326 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39841 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39841: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
I20260929 16:29:47.320152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 17197
I20260929 16:29:47.320343 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:47.320081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.3:51297)
0929 16:29:47.320210 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:47.320273 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 16:29:47.321048 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:47.320806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.3:37855)
0929 16:29:47.320926 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:47.320980 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 16:29:47.321322 17326 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:29:47.975348 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:47.974802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.1:39587)
0929 16:29:47.975144 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:47.975219 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":103,"threads_started":1}
I20260929 16:29:48.132117 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:48.131563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.2:43549)
0929 16:29:48.131917 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:48.131990 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":91,"threads_started":1}
I20260929 16:29:48.301309 17332 raft_consensus.cc:482] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:29:48.301523 17332 raft_consensus.cc:504] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:48.302691 17332 leader_election.cc:288] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2e1c0eed17954a1cac7908f34e3d8b63 (127.0.98.3:37331), 28c94367cb9e4c69b932928e198b4fb2 (127.0.98.2:37725)
Sep 29 16:29:48 dist-test-slave-vq9v krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 29 16:29:48 dist-test-slave-vq9v krb5kdc[15751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260929 16:29:48.322297 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:48.321813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.3:50709)
0929 16:29:48.322122 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:48.322176 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":80,"threads_started":1}
I20260929 16:29:48.322443 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:48.303024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37331 (local address 127.0.98.1:35777)
0929 16:29:48.303598 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:48.303613 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:48.303715 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:48.305525 (+ 1810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:48.305533 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:48.306238 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:48.307249 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:48.307274 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:48.309274 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:48.309278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:48.310931 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:48.310943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:48.311960 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:48.311964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:48.312292 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:48.313067 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:48.313098 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:48.315847 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:48.319895 (+ 4048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:48.319898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:48.319899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:48.320136 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:48.320491 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:48.320494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:48.320501 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:48.320603 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:48.321433 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:48.321439 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:48.321662 (+ 223us) client_negotiation.cc:769] Sending connection context
0929 16:29:48.321856 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 16:29:48.322217 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":580,"spinlock_wait_cycles":19072,"thread_start_us":85,"threads_started":1}
I20260929 16:29:48.322521 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:48.303320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35777 (local address 127.0.98.3:37331)
0929 16:29:48.303746 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:48.303756 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:48.303789 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:29:48.304566 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:48.304575 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:48.305040 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:48.305410 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:48.307480 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:48.309062 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:48.311064 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:48.311812 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:48.311995 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:48.316641 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:48.316649 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:48.316668 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:48.316702 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:48.319761 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:48.320258 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:48.320262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:48.320267 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:48.320375 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:48.320768 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:48.320771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:48.320772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:48.321307 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:48.321449 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:48.322008 (+ 559us) server_negotiation.cc:299] Negotiation successful
0929 16:29:48.322274 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":103,"threads_started":1}
I20260929 16:29:48.322623 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:48.303274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59749 (local address 127.0.98.2:37725)
0929 16:29:48.303582 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0929 16:29:48.303609 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 16:29:48.303652 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:29:48.304566 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:29:48.304575 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:29:48.305040 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:29:48.305410 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:29:48.307480 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:48.309062 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:48.311064 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:29:48.311827 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:29:48.311996 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:48.316075 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:29:48.316080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:29:48.316093 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:29:48.316126 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:29:48.318627 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:48.319327 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:48.319332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:48.319337 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:48.319457 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:29:48.320309 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:29:48.320313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:29:48.320315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:29:48.321282 (+ 967us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:29:48.321420 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 16:29:48.322120 (+ 700us) server_negotiation.cc:299] Negotiation successful
0929 16:29:48.322448 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":125,"threads_started":1}
I20260929 16:29:48.323096 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:48.303024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37725 (local address 127.0.98.1:59749)
0929 16:29:48.303194 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:48.303212 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:29:48.303333 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:29:48.305526 (+ 2193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:29:48.305533 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:29:48.306235 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:29:48.307249 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:48.307274 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:48.309274 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:48.309278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:48.310931 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:29:48.310943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:29:48.311960 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:29:48.311964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:29:48.312291 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:29:48.313081 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:29:48.313100 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:29:48.315832 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:29:48.318774 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:48.318783 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:48.318800 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:48.319177 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:48.319570 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:29:48.319574 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:29:48.319576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:29:48.319767 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:29:48.321401 (+ 1634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:29:48.321407 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:29:48.321739 (+ 332us) client_negotiation.cc:769] Sending connection context
0929 16:29:48.321856 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 16:29:48.322934 (+ 1078us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":1273,"spinlock_wait_cycles":10368}
W20260929 16:29:48.325127 16948 leader_election.cc:341] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28c94367cb9e4c69b932928e198b4fb2 (127.0.98.2:37725): Not found: Tablet not found: 814c316fbda84c7a859b92d60fb97645
W20260929 16:29:48.325364 16947 leader_election.cc:341] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2e1c0eed17954a1cac7908f34e3d8b63 (127.0.98.3:37331): Not found: Tablet not found: 814c316fbda84c7a859b92d60fb97645
I20260929 16:29:48.325445 16947 leader_election.cc:302] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ed5e5c9cc0450783fa75521dff94e4; no voters: 28c94367cb9e4c69b932928e198b4fb2, 2e1c0eed17954a1cac7908f34e3d8b63
I20260929 16:29:48.325673 17332 raft_consensus.cc:2726] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:29:48.976713 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:48.976241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.1:53771)
0929 16:29:48.976527 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:48.976594 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":121,"threads_started":1}
I20260929 16:29:49.133700 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:49.133169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.2:55915)
0929 16:29:49.133515 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:49.133591 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":84,"threads_started":1}
I20260929 16:29:49.324214 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:49.323774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.3:59125)
0929 16:29:49.324037 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:49.324101 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":80,"threads_started":1}
I20260929 16:29:49.978300 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:49.977822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.1:44589)
0929 16:29:49.978111 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:49.978180 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":78,"threads_started":1}
I20260929 16:29:49.995103 17341 raft_consensus.cc:482] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:29:49.995241 17341 raft_consensus.cc:504] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:49.995762 17341 leader_election.cc:288] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2e1c0eed17954a1cac7908f34e3d8b63 (127.0.98.3:37331), 28c94367cb9e4c69b932928e198b4fb2 (127.0.98.2:37725)
W20260929 16:29:49.996853 16948 leader_election.cc:341] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28c94367cb9e4c69b932928e198b4fb2 (127.0.98.2:37725): Not found: Tablet not found: 814c316fbda84c7a859b92d60fb97645
W20260929 16:29:49.996964 16947 leader_election.cc:341] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2e1c0eed17954a1cac7908f34e3d8b63 (127.0.98.3:37331): Not found: Tablet not found: 814c316fbda84c7a859b92d60fb97645
I20260929 16:29:49.997023 16947 leader_election.cc:302] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ed5e5c9cc0450783fa75521dff94e4; no voters: 28c94367cb9e4c69b932928e198b4fb2, 2e1c0eed17954a1cac7908f34e3d8b63
I20260929 16:29:49.997170 17341 raft_consensus.cc:2726] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:29:50.135262 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:50.134756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.2:44479)
0929 16:29:50.135033 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:50.135098 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":111,"threads_started":1}
I20260929 16:29:50.325667 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:50.325135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.3:48733)
0929 16:29:50.325451 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:50.325513 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":132,"threads_started":1}
I20260929 16:29:50.979569 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:50.979177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.1:52379)
0929 16:29:50.979424 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:50.979475 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260929 16:29:51.136482 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:51.136049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.2:59669)
0929 16:29:51.136325 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:51.136387 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":77,"threads_started":1}
I20260929 16:29:51.326993 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:51.326554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.3:34421)
0929 16:29:51.326831 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:51.326894 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
I20260929 16:29:51.980970 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:51.980499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.1:47677)
0929 16:29:51.980815 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:51.980876 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":86,"threads_started":1}
I20260929 16:29:52.137969 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:52.137501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.2:55621)
0929 16:29:52.137774 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:52.137868 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":86,"threads_started":1}
I20260929 16:29:52.328536 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:52.328074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.3:55285)
0929 16:29:52.328371 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:52.328440 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":87,"threads_started":1}
I20260929 16:29:52.975332 17350 raft_consensus.cc:482] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:29:52.975500 17350 raft_consensus.cc:504] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:52.975941 17350 leader_election.cc:288] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2e1c0eed17954a1cac7908f34e3d8b63 (127.0.98.3:37331), 28c94367cb9e4c69b932928e198b4fb2 (127.0.98.2:37725)
W20260929 16:29:52.976889 16948 leader_election.cc:341] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28c94367cb9e4c69b932928e198b4fb2 (127.0.98.2:37725): Not found: Tablet not found: 814c316fbda84c7a859b92d60fb97645
W20260929 16:29:52.977013 16947 leader_election.cc:341] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2e1c0eed17954a1cac7908f34e3d8b63 (127.0.98.3:37331): Not found: Tablet not found: 814c316fbda84c7a859b92d60fb97645
I20260929 16:29:52.977083 16947 leader_election.cc:302] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ed5e5c9cc0450783fa75521dff94e4; no voters: 28c94367cb9e4c69b932928e198b4fb2, 2e1c0eed17954a1cac7908f34e3d8b63
I20260929 16:29:52.977280 17350 raft_consensus.cc:2726] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:29:52.982278 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:52.981837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.1:46139)
0929 16:29:52.982111 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:52.982181 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260929 16:29:53.139576 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:53.139012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.2:52667)
0929 16:29:53.139369 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:53.139433 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":165,"threads_started":1}
I20260929 16:29:53.329846 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:53.329423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.3:51853)
0929 16:29:53.329684 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:53.329747 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
I20260929 16:29:53.983565 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:53.983102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.1:34023)
0929 16:29:53.983395 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:53.983462 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":79,"threads_started":1}
I20260929 16:29:54.140925 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:54.140440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.2:52695)
0929 16:29:54.140762 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:54.140830 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":123,"threads_started":1}
I20260929 16:29:54.331225 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:54.330710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.3:38591)
0929 16:29:54.331067 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:54.331130 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":111,"threads_started":1}
I20260929 16:29:54.984886 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:54.984423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.1:38695)
0929 16:29:54.984718 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:54.984787 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":80,"threads_started":1}
I20260929 16:29:55.142437 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:55.141926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.2:33163)
0929 16:29:55.142254 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:55.142336 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":94,"threads_started":1}
I20260929 16:29:55.332504 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:55.332065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.3:47401)
0929 16:29:55.332344 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:55.332408 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
I20260929 16:29:55.986186 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:55.985742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.1:41925)
0929 16:29:55.986010 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:55.986080 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":104,"threads_started":1}
I20260929 16:29:56.143996 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:56.143498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.2:50779)
0929 16:29:56.143825 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:56.143900 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":104,"threads_started":1}
I20260929 16:29:56.333976 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:56.333500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.3:35823)
0929 16:29:56.333780 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:56.333878 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":96,"threads_started":1}
I20260929 16:29:56.987777 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:56.987269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.1:60523)
0929 16:29:56.987570 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:56.987635 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":97,"threads_started":1}
I20260929 16:29:57.145442 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:57.144933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.2:36593)
0929 16:29:57.145266 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:57.145343 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
I20260929 16:29:57.335479 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:57.334905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.3:40317)
0929 16:29:57.335248 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:57.335354 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":115,"threads_started":1}
I20260929 16:29:57.560124 17366 raft_consensus.cc:482] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:29:57.560269 17366 raft_consensus.cc:504] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:29:57.560802 17366 leader_election.cc:288] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2e1c0eed17954a1cac7908f34e3d8b63 (127.0.98.3:37331), 28c94367cb9e4c69b932928e198b4fb2 (127.0.98.2:37725)
W20260929 16:29:57.561609 16947 leader_election.cc:341] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2e1c0eed17954a1cac7908f34e3d8b63 (127.0.98.3:37331): Not found: Tablet not found: 814c316fbda84c7a859b92d60fb97645
W20260929 16:29:57.561702 16948 leader_election.cc:341] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28c94367cb9e4c69b932928e198b4fb2 (127.0.98.2:37725): Not found: Tablet not found: 814c316fbda84c7a859b92d60fb97645
I20260929 16:29:57.561844 16948 leader_election.cc:302] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ed5e5c9cc0450783fa75521dff94e4; no voters: 28c94367cb9e4c69b932928e198b4fb2, 2e1c0eed17954a1cac7908f34e3d8b63
I20260929 16:29:57.562074 17366 raft_consensus.cc:2726] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:29:57.989423 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:57.988835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.1:60731)
0929 16:29:57.989181 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:57.989272 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":111,"threads_started":1}
I20260929 16:29:58.146751 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:58.146293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.2:39509)
0929 16:29:58.146570 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:58.146652 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":73,"threads_started":1}
I20260929 16:29:58.336688 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:58.336269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.3:49717)
0929 16:29:58.336533 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:58.336592 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":72,"threads_started":1}
I20260929 16:29:58.990664 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:58.990245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.1:41575)
0929 16:29:58.990510 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:58.990566 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":89,"threads_started":1}
I20260929 16:29:59.148178 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:59.147733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.2:55071)
0929 16:29:59.148016 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:59.148079 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":92,"threads_started":1}
I20260929 16:29:59.338080 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:59.337643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.3:40623)
0929 16:29:59.337914 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:59.337980 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260929 16:29:59.992007 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:29:59.991558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.1:41387)
0929 16:29:59.991841 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 16:29:59.991910 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":82,"threads_started":1}
I20260929 16:30:00.149845 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:00.149278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.2:57621)
0929 16:30:00.149601 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:00.149677 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":91,"threads_started":1}
I20260929 16:30:00.339395 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:00.338968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.3:42139)
0929 16:30:00.339232 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:00.339292 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":69,"threads_started":1}
I20260929 16:30:00.993515 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:00.992945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.1:57639)
0929 16:30:00.993329 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:00.993413 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":92,"threads_started":1}
I20260929 16:30:01.151329 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:01.150850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.2:54113)
0929 16:30:01.151157 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:01.151214 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":99,"threads_started":1}
I20260929 16:30:01.340608 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:01.340197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.3:37605)
0929 16:30:01.340453 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:01.340510 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":87,"threads_started":1}
I20260929 16:30:01.540623 17379 raft_consensus.cc:482] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:30:01.540774 17379 raft_consensus.cc:504] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1c0eed17954a1cac7908f34e3d8b63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37331 } } peers { permanent_uuid: "28c94367cb9e4c69b932928e198b4fb2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37725 } } peers { permanent_uuid: "31ed5e5c9cc0450783fa75521dff94e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46609 } }
I20260929 16:30:01.541375 17379 leader_election.cc:288] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2e1c0eed17954a1cac7908f34e3d8b63 (127.0.98.3:37331), 28c94367cb9e4c69b932928e198b4fb2 (127.0.98.2:37725)
W20260929 16:30:01.542323 16948 leader_election.cc:341] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28c94367cb9e4c69b932928e198b4fb2 (127.0.98.2:37725): Not found: Tablet not found: 814c316fbda84c7a859b92d60fb97645
W20260929 16:30:01.542469 16947 leader_election.cc:341] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2e1c0eed17954a1cac7908f34e3d8b63 (127.0.98.3:37331): Not found: Tablet not found: 814c316fbda84c7a859b92d60fb97645
I20260929 16:30:01.542534 16947 leader_election.cc:302] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31ed5e5c9cc0450783fa75521dff94e4; no voters: 28c94367cb9e4c69b932928e198b4fb2, 2e1c0eed17954a1cac7908f34e3d8b63
I20260929 16:30:01.542689 17379 raft_consensus.cc:2726] T 814c316fbda84c7a859b92d60fb97645 P 31ed5e5c9cc0450783fa75521dff94e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:30:01.994947 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:01.994531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.1:60765)
0929 16:30:01.994793 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:01.994849 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":101,"threads_started":1}
I20260929 16:30:02.152782 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:02.152382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39841 (local address 127.0.98.2:37475)
0929 16:30:02.152623 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:02.152687 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":94,"threads_started":1}
I20260929 16:30:02.329170 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 16897
I20260929 16:30:02.335602 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 17063
I20260929 16:30:02.341581 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 17197
2026-09-29T16:30:02Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59304 ms)
[----------] 30 tests from SecurityITest (249938 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:02 dist-test-slave-vq9v krb5kdc[17384](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:02 dist-test-slave-vq9v krb5kdc[17384](info): set up 2 sockets
Sep 29 16:30:02 dist-test-slave-vq9v krb5kdc[17384](info): commencing operation
krb5kdc: starting...
W20260929 16:30:03.441057 392 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:03 dist-test-slave-vq9v krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:30:03.578042 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44253
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:03.699118 17397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:03.699429 17397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:03.699473 17397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:03.703819 17397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:03.703907 17397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:03.703928 17397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:03.703945 17397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:03.703962 17397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:03.709755 17397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:03.710958 17397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:03.712071 17397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:03.721534 17403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:03.721513 17402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:03.721513 17405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:03.721633 17397 server_base.cc:1034] running on GCE node
I20260929 16:30:03.722549 17397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:03.723839 17397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:03.725086 17397 hybrid_clock.cc:648] HybridClock initialized: now 1790699403725055 us; error 58 us; skew 500 ppm
Sep 29 16:30:03 dist-test-slave-vq9v krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:03.732630 17397 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:30:03.733956 17397 webserver.cc:460] Webserver started at http://127.0.98.62:46591/ using document root <none> and password file <none>
I20260929 16:30:03.734623 17397 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:03.734738 17397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:03.734999 17397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:03.737466 17397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "2e3f9025719243b690ec8645e92cec4c"
format_stamp: "Formatted at 2026-09-29 16:30:03 on dist-test-slave-vq9v"
server_key: "8c266d7149f7eb566a274015bbed8c85"
server_key_iv: "83a76c5f78804d814b274d36ebadf934"
server_key_version: "encryptionkey@0"
I20260929 16:30:03.738116 17397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e3f9025719243b690ec8645e92cec4c"
format_stamp: "Formatted at 2026-09-29 16:30:03 on dist-test-slave-vq9v"
server_key: "8c266d7149f7eb566a274015bbed8c85"
server_key_iv: "83a76c5f78804d814b274d36ebadf934"
server_key_version: "encryptionkey@0"
I20260929 16:30:03.742477 17397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:30:03.745555 17412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:03.747047 17397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:30:03.747243 17397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "2e3f9025719243b690ec8645e92cec4c"
format_stamp: "Formatted at 2026-09-29 16:30:03 on dist-test-slave-vq9v"
server_key: "8c266d7149f7eb566a274015bbed8c85"
server_key_iv: "83a76c5f78804d814b274d36ebadf934"
server_key_version: "encryptionkey@0"
I20260929 16:30:03.747395 17397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:03.761313 17397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:03.764956 17397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:03.765323 17397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:03.773067 17397 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44253
I20260929 16:30:03.773164 17464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44253 every 8 connection(s)
I20260929 16:30:03.774236 17397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:03.775627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 17397
I20260929 16:30:03.775781 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:03.776078 392 external_mini_cluster.cc:1428] Setting key a60c475b63ddc17c400d6a3f91c7a6af
I20260929 16:30:03.777616 17465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:03.784508 17465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c: Bootstrap starting.
Sep 29 16:30:03 dist-test-slave-vq9v krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:30:03.786995 17465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:03.787760 17465 log.cc:824] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:03.790125 17465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c: No bootstrap required, opened a new log
I20260929 16:30:03.791530 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:03.777494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37116 (local address 127.0.98.62:44253)
0929 16:30:03.777939 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:03.777954 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:03.777990 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:30:03.778620 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:03.778636 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:03.778964 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:03.779267 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:03.780158 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:03.781173 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:03.782395 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:03.782863 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:03.782982 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:03.785910 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:03.785916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:03.785930 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:03.785961 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:03.787827 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:03.788323 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:03.788329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:03.788336 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:03.788464 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:03.788772 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:03.788776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:03.788777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:03.789442 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:03.789662 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:03.789991 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 16:30:03.790369 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":150,"threads_started":1}
I20260929 16:30:03.794135 17465 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3f9025719243b690ec8645e92cec4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } }
I20260929 16:30:03.794474 17465 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:03.794561 17465 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e3f9025719243b690ec8645e92cec4c, State: Initialized, Role: FOLLOWER
I20260929 16:30:03.795051 17465 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3f9025719243b690ec8645e92cec4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } }
I20260929 16:30:03.795208 17465 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:03.795292 17465 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:03.795404 17465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:03.796478 17465 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3f9025719243b690ec8645e92cec4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } }
I20260929 16:30:03.796876 17465 leader_election.cc:302] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e3f9025719243b690ec8645e92cec4c; no voters:
I20260929 16:30:03.797312 17465 leader_election.cc:288] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:03.797392 17470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:03.797640 17470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c [term 1 LEADER]: Becoming Leader. State: Replica: 2e3f9025719243b690ec8645e92cec4c, State: Running, Role: LEADER
I20260929 16:30:03.797960 17470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3f9025719243b690ec8645e92cec4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } }
I20260929 16:30:03.798657 17465 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:03.799669 17471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e3f9025719243b690ec8645e92cec4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3f9025719243b690ec8645e92cec4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } } }
I20260929 16:30:03.799855 17471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:03.800149 17472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e3f9025719243b690ec8645e92cec4c. Latest consensus state: current_term: 1 leader_uuid: "2e3f9025719243b690ec8645e92cec4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3f9025719243b690ec8645e92cec4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } } }
I20260929 16:30:03.800262 17472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:03.800277 17479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:03.803341 17479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:03.810751 17479 catalog_manager.cc:1347] Generated new cluster ID: e8bc7c9bd8764c9c8d8544362d4a5c13
I20260929 16:30:03.810838 17479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:03.821834 17479 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:03.823124 17479 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:03.829734 17479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c: Generated new TSK 0
I20260929 16:30:03.830662 17479 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 16:30:03.873358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44253
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:03.993728 17491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:03.994048 17491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:03.994094 17491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:03.998468 17491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:03.998546 17491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:03.998615 17491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:30:04.004829 17491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:04.006084 17491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:04.007123 17491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:04.017405 17497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:04.017452 17499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:04.017452 17496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:04.017946 17491 server_base.cc:1034] running on GCE node
I20260929 16:30:04.018421 17491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:04.019078 17491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:04.020274 17491 hybrid_clock.cc:648] HybridClock initialized: now 1790699404020249 us; error 38 us; skew 500 ppm
Sep 29 16:30:04 dist-test-slave-vq9v krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:04.023715 17491 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:30:04.024842 17491 webserver.cc:460] Webserver started at http://127.0.98.1:43919/ using document root <none> and password file <none>
I20260929 16:30:04.025667 17491 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:04.025792 17491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:04.026049 17491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:04.028345 17491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "5b1e8c6e1a5a4fc09ad341c53df60fce"
format_stamp: "Formatted at 2026-09-29 16:30:04 on dist-test-slave-vq9v"
server_key: "8ccfd8bb752c98ff99eece092c916b26"
server_key_iv: "54246fd4753457ca7c183e1a6b641c19"
server_key_version: "encryptionkey@0"
I20260929 16:30:04.028995 17491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5b1e8c6e1a5a4fc09ad341c53df60fce"
format_stamp: "Formatted at 2026-09-29 16:30:04 on dist-test-slave-vq9v"
server_key: "8ccfd8bb752c98ff99eece092c916b26"
server_key_iv: "54246fd4753457ca7c183e1a6b641c19"
server_key_version: "encryptionkey@0"
I20260929 16:30:04.033166 17491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260929 16:30:04.036149 17506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:04.037802 17491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:04.037992 17491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "5b1e8c6e1a5a4fc09ad341c53df60fce"
format_stamp: "Formatted at 2026-09-29 16:30:04 on dist-test-slave-vq9v"
server_key: "8ccfd8bb752c98ff99eece092c916b26"
server_key_iv: "54246fd4753457ca7c183e1a6b641c19"
server_key_version: "encryptionkey@0"
I20260929 16:30:04.038156 17491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:04.054901 17491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:04.058836 17491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:04.059080 17491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:04.059762 17491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:04.060951 17491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:04.061026 17491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:04.061100 17491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:04.061136 17491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:04.072204 17491 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45045
I20260929 16:30:04.072230 17619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45045 every 8 connection(s)
I20260929 16:30:04.073316 17491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:30:04.080842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 17491
I20260929 16:30:04.080977 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:30:04.081408 392 external_mini_cluster.cc:1428] Setting key a6e5f2915f06b2d5b3c4e42306bb410c
Sep 29 16:30:04 dist-test-slave-vq9v krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:30:04.091452 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:04.075416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32899 (local address 127.0.98.62:44253)
0929 16:30:04.075596 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:04.075601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:04.076558 (+ 957us) server_negotiation.cc:407] Connection header received
0929 16:30:04.078688 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:04.078691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:04.078747 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:04.078839 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:04.080337 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.081028 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.082642 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.082926 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.083060 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.086214 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:04.086222 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:04.086229 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:04.086269 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:04.088470 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:04.089192 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:04.089198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:04.089251 (+ 53us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:04.089373 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:04.089865 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:04.089872 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:04.089877 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:04.090171 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:04.090305 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:04.091052 (+ 747us) server_negotiation.cc:299] Negotiation successful
0929 16:30:04.091226 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:30:04.092337 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:04.075816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44253 (local address 127.0.98.1:32899)
0929 16:30:04.076373 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:04.076425 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:04.078403 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:04.079045 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:04.079053 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:04.079496 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:04.080062 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:04.080078 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.081168 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.081174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:04.082498 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:04.082509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.083105 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.083109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:04.083348 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.084111 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:04.084130 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:04.085984 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:04.088648 (+ 2664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:04.088655 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:04.088703 (+ 48us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:04.089037 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:04.089516 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:04.089519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:04.089520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:04.089733 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:04.090342 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:04.090349 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:04.090708 (+ 359us) client_negotiation.cc:769] Sending connection context
0929 16:30:04.090922 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 16:30:04.091265 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":137,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 16:30:04.094132 17620 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44253
I20260929 16:30:04.094440 17620 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:04.095062 17620 heartbeater.cc:511] Master 127.0.98.62:44253 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 16:30:04.097092 17429 ts_manager.cc:194] Registered new tserver with Master: 5b1e8c6e1a5a4fc09ad341c53df60fce (127.0.98.1:45045)
I20260929 16:30:04.098690 17429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 16:30:04.118067 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44253
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:04.244930 17625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:04.245326 17625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:04.245381 17625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:04.250059 17625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:04.250139 17625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:04.250213 17625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:30:04.256657 17625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:04.257957 17625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:04.259083 17625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:04.270833 17631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:04.270833 17633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:04.270818 17630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:04.271245 17625 server_base.cc:1034] running on GCE node
I20260929 16:30:04.271685 17625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:04.272423 17625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:04.273597 17625 hybrid_clock.cc:648] HybridClock initialized: now 1790699404273550 us; error 57 us; skew 500 ppm
Sep 29 16:30:04 dist-test-slave-vq9v krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:04.276477 17625 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:30:04.277535 17625 webserver.cc:460] Webserver started at http://127.0.98.2:36533/ using document root <none> and password file <none>
I20260929 16:30:04.278126 17625 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:04.278220 17625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:04.278461 17625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:04.280637 17625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "ea46ce661b18413bb4b81e3a4317643f"
format_stamp: "Formatted at 2026-09-29 16:30:04 on dist-test-slave-vq9v"
server_key: "08721c5c57a71e93146ea80b61b96b2d"
server_key_iv: "772afc06204ad91e8a0f7fb748f5ff14"
server_key_version: "encryptionkey@0"
I20260929 16:30:04.281297 17625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ea46ce661b18413bb4b81e3a4317643f"
format_stamp: "Formatted at 2026-09-29 16:30:04 on dist-test-slave-vq9v"
server_key: "08721c5c57a71e93146ea80b61b96b2d"
server_key_iv: "772afc06204ad91e8a0f7fb748f5ff14"
server_key_version: "encryptionkey@0"
I20260929 16:30:04.285351 17625 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:30:04.287952 17640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:04.289453 17625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:04.289590 17625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "ea46ce661b18413bb4b81e3a4317643f"
format_stamp: "Formatted at 2026-09-29 16:30:04 on dist-test-slave-vq9v"
server_key: "08721c5c57a71e93146ea80b61b96b2d"
server_key_iv: "772afc06204ad91e8a0f7fb748f5ff14"
server_key_version: "encryptionkey@0"
I20260929 16:30:04.289716 17625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:04.330667 17625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:04.334317 17625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:04.334506 17625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:04.335249 17625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:04.336139 17625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:04.336190 17625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:04.336294 17625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:04.336321 17625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:04.345719 17625 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33915
I20260929 16:30:04.345757 17753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33915 every 8 connection(s)
I20260929 16:30:04.346720 17625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:30:04.356820 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 17625
I20260929 16:30:04.356966 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
Sep 29 16:30:04 dist-test-slave-vq9v krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:30:04.357333 392 external_mini_cluster.cc:1428] Setting key 225836767d8d34b93e4482214b934107
I20260929 16:30:04.362411 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:04.348700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49689 (local address 127.0.98.62:44253)
0929 16:30:04.348863 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:04.348867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:04.349885 (+ 1018us) server_negotiation.cc:407] Connection header received
0929 16:30:04.351322 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:04.351326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:04.351386 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:04.351498 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:04.352804 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.353400 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.354577 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.354809 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.354919 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.358007 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:04.358016 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:04.358019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:04.358047 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:04.359840 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:04.360389 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:04.360396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:04.360400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:04.360501 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:04.360949 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:04.360954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:04.360957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:04.361248 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:04.361332 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:04.362167 (+ 835us) server_negotiation.cc:299] Negotiation successful
0929 16:30:04.362286 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:30:04.363693 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:04.349162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44253 (local address 127.0.98.2:49689)
0929 16:30:04.349724 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:04.349770 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:04.351049 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:04.351697 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:04.351705 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:04.352090 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:04.352648 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:04.352662 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.353534 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.353537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:04.354447 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:04.354456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.354964 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.354968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:04.355145 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.355790 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:04.355806 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:04.357783 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:04.359968 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:04.359973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:04.359996 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:04.360279 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:04.360595 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:04.360597 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:04.360598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:04.360811 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:04.361425 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:04.361432 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:04.361847 (+ 415us) client_negotiation.cc:769] Sending connection context
0929 16:30:04.362098 (+ 251us) client_negotiation.cc:240] Negotiation successful
0929 16:30:04.362496 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":137,"threads_started":1}
I20260929 16:30:04.365036 17754 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44253
I20260929 16:30:04.365377 17754 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:04.366040 17754 heartbeater.cc:511] Master 127.0.98.62:44253 requested a full tablet report, sending...
I20260929 16:30:04.367343 17429 ts_manager.cc:194] Registered new tserver with Master: ea46ce661b18413bb4b81e3a4317643f (127.0.98.2:33915)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 16:30:04.368078 17429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 16:30:04.388947 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44253
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:04.504642 17759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:04.504922 17759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:04.504998 17759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:04.509451 17759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:04.509529 17759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:04.509593 17759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:30:04.515359 17759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:04.516526 17759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:04.517697 17759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:04.528012 17767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:04.528126 17759 server_base.cc:1034] running on GCE node
W20260929 16:30:04.527980 17765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:04.528005 17764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:04.528820 17759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:04.529593 17759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:04.530828 17759 hybrid_clock.cc:648] HybridClock initialized: now 1790699404530783 us; error 70 us; skew 500 ppm
Sep 29 16:30:04 dist-test-slave-vq9v krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:04.534017 17759 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:30:04.535113 17759 webserver.cc:460] Webserver started at http://127.0.98.3:43037/ using document root <none> and password file <none>
I20260929 16:30:04.535724 17759 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:04.535832 17759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:04.536072 17759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:04.538302 17759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "6c5b0c1acfa54c58a63757838f2cdeb6"
format_stamp: "Formatted at 2026-09-29 16:30:04 on dist-test-slave-vq9v"
server_key: "5752977489343bb7173156a19f777784"
server_key_iv: "591650ef7819994627867bd6f67e9f17"
server_key_version: "encryptionkey@0"
I20260929 16:30:04.538887 17759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6c5b0c1acfa54c58a63757838f2cdeb6"
format_stamp: "Formatted at 2026-09-29 16:30:04 on dist-test-slave-vq9v"
server_key: "5752977489343bb7173156a19f777784"
server_key_iv: "591650ef7819994627867bd6f67e9f17"
server_key_version: "encryptionkey@0"
I20260929 16:30:04.542850 17759 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:30:04.545631 17774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:04.547024 17759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:30:04.547178 17759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "6c5b0c1acfa54c58a63757838f2cdeb6"
format_stamp: "Formatted at 2026-09-29 16:30:04 on dist-test-slave-vq9v"
server_key: "5752977489343bb7173156a19f777784"
server_key_iv: "591650ef7819994627867bd6f67e9f17"
server_key_version: "encryptionkey@0"
I20260929 16:30:04.547312 17759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:04.559150 17759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:04.562673 17759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:04.562889 17759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:04.563478 17759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:04.564316 17759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:04.564384 17759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:04.564455 17759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:04.564514 17759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:04.574184 17759 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40829
I20260929 16:30:04.574208 17887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40829 every 8 connection(s)
I20260929 16:30:04.575198 17759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:30:04.575570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 17759
I20260929 16:30:04.575708 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:30:04.576062 392 external_mini_cluster.cc:1428] Setting key 7d78bd5ea31e119d3d1b7c8bb55d5dae
Sep 29 16:30:04 dist-test-slave-vq9v krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:30:04.590436 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:04.577393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43853 (local address 127.0.98.62:44253)
0929 16:30:04.577547 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:04.577550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:04.578525 (+ 975us) server_negotiation.cc:407] Connection header received
0929 16:30:04.579939 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:04.579941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:04.579986 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:04.580054 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:04.581417 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.581947 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.583192 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.583425 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.583530 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.586274 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:04.586278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:04.586280 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:04.586307 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:04.587813 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:04.588466 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:04.588469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:04.588471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:04.588558 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:04.589111 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:04.589121 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:04.589122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:04.589413 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:04.589538 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:04.590120 (+ 582us) server_negotiation.cc:299] Negotiation successful
0929 16:30:04.590261 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:30:04.591373 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:04.577843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44253 (local address 127.0.98.3:43853)
0929 16:30:04.578355 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:04.578413 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:04.579740 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:04.580239 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:04.580247 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:04.580628 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:04.581265 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:04.581289 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.582057 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.582059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:04.583042 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:04.583051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.583580 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.583583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:04.583777 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.584371 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:04.584388 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:04.586079 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:04.587944 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:04.587950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:04.587965 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:04.588298 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:04.588692 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:04.588696 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:04.588698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:04.588870 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:04.589538 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:04.589547 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:04.589849 (+ 302us) client_negotiation.cc:769] Sending connection context
0929 16:30:04.590062 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 16:30:04.590456 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":154,"threads_started":1}
I20260929 16:30:04.592552 17888 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44253
I20260929 16:30:04.592866 17888 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:04.593551 17888 heartbeater.cc:511] Master 127.0.98.62:44253 requested a full tablet report, sending...
I20260929 16:30:04.594688 17429 ts_manager.cc:194] Registered new tserver with Master: 6c5b0c1acfa54c58a63757838f2cdeb6 (127.0.98.3:40829)
I20260929 16:30:04.595299 17429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43853
I20260929 16:30:04.602690 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:30:04.612884 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:04.604832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37120 (local address 127.0.98.62:44253)
0929 16:30:04.605013 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:04.605019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:04.605156 (+ 137us) server_negotiation.cc:407] Connection header received
0929 16:30:04.605322 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:04.605325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:04.605371 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:04.605446 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:04.606192 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.606783 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.607638 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.607892 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.607982 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.609647 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:04.609651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:04.609653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:04.609681 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:04.611202 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:04.611674 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:04.611677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:04.611678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:04.611762 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:04.612017 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:04.612020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:04.612021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:04.612278 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:04.612380 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:04.612602 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 16:30:04.612701 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:30:04.616230 17429 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:30:04.619601 17429 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:30:04.634887 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:04.628496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40829 (local address 127.0.0.1:43034)
0929 16:30:04.629371 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:04.629427 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:04.629564 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:04.630470 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:04.630477 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:04.630516 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:04.630821 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:04.630831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.632003 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.632008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:04.633252 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:04.633265 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.634258 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.634261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:04.634400 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.634443 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 16:30:04.634566 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 16:30:04.634718 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":85,"spinlock_wait_cycles":8576,"thread_start_us":101,"threads_started":1}
I20260929 16:30:04.634897 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:04.628670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43034 (local address 127.0.98.3:40829)
0929 16:30:04.629899 (+ 1229us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:04.629912 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:04.629936 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:30:04.630014 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:04.630019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:04.630234 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:04.630400 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:04.630955 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.631875 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.633381 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.634130 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.634246 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.634408 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:04.634615 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 16:30:04.634716 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1138,"thread_start_us":195,"threads_started":1}
I20260929 16:30:04.635389 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:04.628911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45045 (local address 127.0.0.1:50632)
0929 16:30:04.630633 (+ 1722us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:04.630644 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:04.630758 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:04.631244 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:04.631247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:04.631265 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:04.631353 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:04.631358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.632722 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.632725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:04.633791 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:04.633798 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.634945 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.634951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:04.635061 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.635095 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:30:04.635182 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 16:30:04.635247 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1668,"thread_start_us":89,"threads_started":1}
I20260929 16:30:04.635597 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:04.629398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55242 (local address 127.0.98.2:33915)
0929 16:30:04.629899 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:04.629912 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:04.629936 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:30:04.630014 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:04.630019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:04.630234 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:04.630400 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:04.630914 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.632124 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.633662 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.634506 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.634630 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.634755 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:04.635319 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 16:30:04.635409 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":113,"threads_started":1}
I20260929 16:30:04.635258 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:04.628675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33915 (local address 127.0.0.1:55242)
0929 16:30:04.629765 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:04.629780 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:04.629867 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:04.630470 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:04.630477 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:04.630515 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:04.630772 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:04.630784 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.632286 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.632289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:04.633534 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:04.633544 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.634820 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.634824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:04.634922 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.634953 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:30:04.635025 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 16:30:04.635094 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"mutex_wait_us":95,"spinlock_wait_cycles":4736,"thread_start_us":110,"threads_started":1}
I20260929 16:30:04.636135 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:04.629376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50632 (local address 127.0.98.1:45045)
0929 16:30:04.630115 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:04.630120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:04.630741 (+ 621us) server_negotiation.cc:407] Connection header received
0929 16:30:04.630895 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:04.630900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:04.631091 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:04.631242 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:04.631474 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.632575 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.633908 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.634655 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.634753 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.634881 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:04.635217 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 16:30:04.635970 (+ 753us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":88,"threads_started":1}
I20260929 16:30:04.638710 17822 tablet_service.cc:1467] Processing CreateTablet for tablet f826a22f117e4742bc08965c6bfb0d16 (DEFAULT_TABLE table=test-table [id=78fdb0807fa34f889f4670aad910679e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:04.639108 17554 tablet_service.cc:1467] Processing CreateTablet for tablet f826a22f117e4742bc08965c6bfb0d16 (DEFAULT_TABLE table=test-table [id=78fdb0807fa34f889f4670aad910679e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:04.639989 17822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f826a22f117e4742bc08965c6bfb0d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:04.640038 17554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f826a22f117e4742bc08965c6bfb0d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:04.641528 17688 tablet_service.cc:1467] Processing CreateTablet for tablet f826a22f117e4742bc08965c6bfb0d16 (DEFAULT_TABLE table=test-table [id=78fdb0807fa34f889f4670aad910679e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:04.642491 17688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f826a22f117e4742bc08965c6bfb0d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:04.648061 17905 tablet_bootstrap.cc:492] T f826a22f117e4742bc08965c6bfb0d16 P 5b1e8c6e1a5a4fc09ad341c53df60fce: Bootstrap starting.
I20260929 16:30:04.648582 17906 tablet_bootstrap.cc:492] T f826a22f117e4742bc08965c6bfb0d16 P 6c5b0c1acfa54c58a63757838f2cdeb6: Bootstrap starting.
I20260929 16:30:04.649315 17907 tablet_bootstrap.cc:492] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f: Bootstrap starting.
I20260929 16:30:04.651434 17906 tablet_bootstrap.cc:654] T f826a22f117e4742bc08965c6bfb0d16 P 6c5b0c1acfa54c58a63757838f2cdeb6: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:04.651594 17905 tablet_bootstrap.cc:654] T f826a22f117e4742bc08965c6bfb0d16 P 5b1e8c6e1a5a4fc09ad341c53df60fce: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:04.651594 17907 tablet_bootstrap.cc:654] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:04.652631 17905 log.cc:824] T f826a22f117e4742bc08965c6bfb0d16 P 5b1e8c6e1a5a4fc09ad341c53df60fce: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:04.652933 17907 log.cc:824] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:04.653541 17906 log.cc:824] T f826a22f117e4742bc08965c6bfb0d16 P 6c5b0c1acfa54c58a63757838f2cdeb6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:04.655135 17907 tablet_bootstrap.cc:492] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f: No bootstrap required, opened a new log
I20260929 16:30:04.655135 17905 tablet_bootstrap.cc:492] T f826a22f117e4742bc08965c6bfb0d16 P 5b1e8c6e1a5a4fc09ad341c53df60fce: No bootstrap required, opened a new log
I20260929 16:30:04.655362 17907 ts_tablet_manager.cc:1397] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:30:04.655362 17905 ts_tablet_manager.cc:1397] T f826a22f117e4742bc08965c6bfb0d16 P 5b1e8c6e1a5a4fc09ad341c53df60fce: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 16:30:04.655539 17906 tablet_bootstrap.cc:492] T f826a22f117e4742bc08965c6bfb0d16 P 6c5b0c1acfa54c58a63757838f2cdeb6: No bootstrap required, opened a new log
I20260929 16:30:04.655784 17906 ts_tablet_manager.cc:1397] T f826a22f117e4742bc08965c6bfb0d16 P 6c5b0c1acfa54c58a63757838f2cdeb6: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260929 16:30:04.659019 17906 raft_consensus.cc:348] T f826a22f117e4742bc08965c6bfb0d16 P 6c5b0c1acfa54c58a63757838f2cdeb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5b0c1acfa54c58a63757838f2cdeb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "ea46ce661b18413bb4b81e3a4317643f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33915 } } peers { permanent_uuid: "5b1e8c6e1a5a4fc09ad341c53df60fce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45045 } }
I20260929 16:30:04.659334 17906 raft_consensus.cc:374] T f826a22f117e4742bc08965c6bfb0d16 P 6c5b0c1acfa54c58a63757838f2cdeb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:04.659416 17906 raft_consensus.cc:729] T f826a22f117e4742bc08965c6bfb0d16 P 6c5b0c1acfa54c58a63757838f2cdeb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c5b0c1acfa54c58a63757838f2cdeb6, State: Initialized, Role: FOLLOWER
I20260929 16:30:04.659597 17907 raft_consensus.cc:348] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5b0c1acfa54c58a63757838f2cdeb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "ea46ce661b18413bb4b81e3a4317643f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33915 } } peers { permanent_uuid: "5b1e8c6e1a5a4fc09ad341c53df60fce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45045 } }
I20260929 16:30:04.660023 17907 raft_consensus.cc:374] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:04.660128 17907 raft_consensus.cc:729] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea46ce661b18413bb4b81e3a4317643f, State: Initialized, Role: FOLLOWER
I20260929 16:30:04.660024 17905 raft_consensus.cc:348] T f826a22f117e4742bc08965c6bfb0d16 P 5b1e8c6e1a5a4fc09ad341c53df60fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5b0c1acfa54c58a63757838f2cdeb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "ea46ce661b18413bb4b81e3a4317643f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33915 } } peers { permanent_uuid: "5b1e8c6e1a5a4fc09ad341c53df60fce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45045 } }
I20260929 16:30:04.660295 17905 raft_consensus.cc:374] T f826a22f117e4742bc08965c6bfb0d16 P 5b1e8c6e1a5a4fc09ad341c53df60fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:04.660351 17905 raft_consensus.cc:729] T f826a22f117e4742bc08965c6bfb0d16 P 5b1e8c6e1a5a4fc09ad341c53df60fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1e8c6e1a5a4fc09ad341c53df60fce, State: Initialized, Role: FOLLOWER
I20260929 16:30:04.660619 17907 consensus_queue.cc:260] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5b0c1acfa54c58a63757838f2cdeb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "ea46ce661b18413bb4b81e3a4317643f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33915 } } peers { permanent_uuid: "5b1e8c6e1a5a4fc09ad341c53df60fce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45045 } }
I20260929 16:30:04.660681 17905 consensus_queue.cc:260] T f826a22f117e4742bc08965c6bfb0d16 P 5b1e8c6e1a5a4fc09ad341c53df60fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5b0c1acfa54c58a63757838f2cdeb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "ea46ce661b18413bb4b81e3a4317643f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33915 } } peers { permanent_uuid: "5b1e8c6e1a5a4fc09ad341c53df60fce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45045 } }
I20260929 16:30:04.660802 17906 consensus_queue.cc:260] T f826a22f117e4742bc08965c6bfb0d16 P 6c5b0c1acfa54c58a63757838f2cdeb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5b0c1acfa54c58a63757838f2cdeb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "ea46ce661b18413bb4b81e3a4317643f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33915 } } peers { permanent_uuid: "5b1e8c6e1a5a4fc09ad341c53df60fce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45045 } }
I20260929 16:30:04.661568 17888 heartbeater.cc:503] Master 127.0.98.62:44253 was elected leader, sending a full tablet report...
I20260929 16:30:04.661877 17907 ts_tablet_manager.cc:1428] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260929 16:30:04.661877 17905 ts_tablet_manager.cc:1428] T f826a22f117e4742bc08965c6bfb0d16 P 5b1e8c6e1a5a4fc09ad341c53df60fce: Time spent starting tablet: real 0.006s user 0.001s sys 0.007s
I20260929 16:30:04.661934 17906 ts_tablet_manager.cc:1428] T f826a22f117e4742bc08965c6bfb0d16 P 6c5b0c1acfa54c58a63757838f2cdeb6: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260929 16:30:04.662288 17620 heartbeater.cc:503] Master 127.0.98.62:44253 was elected leader, sending a full tablet report...
I20260929 16:30:04.663044 17754 heartbeater.cc:503] Master 127.0.98.62:44253 was elected leader, sending a full tablet report...
I20260929 16:30:04.803399 17911 raft_consensus.cc:482] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:30:04.803663 17911 raft_consensus.cc:504] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5b0c1acfa54c58a63757838f2cdeb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "ea46ce661b18413bb4b81e3a4317643f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33915 } } peers { permanent_uuid: "5b1e8c6e1a5a4fc09ad341c53df60fce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45045 } }
I20260929 16:30:04.804986 17911 leader_election.cc:288] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c5b0c1acfa54c58a63757838f2cdeb6 (127.0.98.3:40829), 5b1e8c6e1a5a4fc09ad341c53df60fce (127.0.98.1:45045)
I20260929 16:30:04.809130 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:04.805476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48521 (local address 127.0.98.1:45045)
0929 16:30:04.805608 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:04.805613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:04.805905 (+ 292us) server_negotiation.cc:407] Connection header received
0929 16:30:04.805991 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:04.805994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:04.806046 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:04.806124 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:04.806434 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.806934 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.808082 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.808532 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.808657 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.808725 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:04.808911 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 16:30:04.808990 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:30:04.809146 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:04.805391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45045 (local address 127.0.98.2:48521)
0929 16:30:04.805793 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:04.805808 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:04.805905 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:04.806145 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:04.806147 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:04.806164 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:04.806322 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:04.806328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.807046 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.807050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:04.807948 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:04.807958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.808663 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.808667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:04.808775 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.808803 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 16:30:04.808895 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 16:30:04.808975 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":88,"thread_start_us":94,"threads_started":1}
I20260929 16:30:04.809335 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:04.805391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40829 (local address 127.0.98.2:43707)
0929 16:30:04.805607 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:04.805624 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:04.805734 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:04.806031 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:04.806034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:04.806054 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:04.806243 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:04.806250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.807077 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.807084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:04.808053 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:04.808063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.808785 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.808789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:04.808927 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.808946 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 16:30:04.809042 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 16:30:04.809125 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":141}
I20260929 16:30:04.809317 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:04.805479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43707 (local address 127.0.98.3:40829)
0929 16:30:04.805678 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:04.805683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:04.805704 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:30:04.805856 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:04.805858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:04.805905 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:04.806018 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:04.806364 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.806955 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.808200 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.808653 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.808786 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.808857 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:04.809059 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 16:30:04.809135 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260929 16:30:04.810076 17842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f826a22f117e4742bc08965c6bfb0d16" candidate_uuid: "ea46ce661b18413bb4b81e3a4317643f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5b0c1acfa54c58a63757838f2cdeb6" is_pre_election: true
I20260929 16:30:04.810081 17574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f826a22f117e4742bc08965c6bfb0d16" candidate_uuid: "ea46ce661b18413bb4b81e3a4317643f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1e8c6e1a5a4fc09ad341c53df60fce" is_pre_election: true
I20260929 16:30:04.810408 17842 raft_consensus.cc:2454] T f826a22f117e4742bc08965c6bfb0d16 P 6c5b0c1acfa54c58a63757838f2cdeb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea46ce661b18413bb4b81e3a4317643f in term 0.
I20260929 16:30:04.810429 17574 raft_consensus.cc:2454] T f826a22f117e4742bc08965c6bfb0d16 P 5b1e8c6e1a5a4fc09ad341c53df60fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea46ce661b18413bb4b81e3a4317643f in term 0.
I20260929 16:30:04.810925 17641 leader_election.cc:302] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c5b0c1acfa54c58a63757838f2cdeb6, ea46ce661b18413bb4b81e3a4317643f; no voters:
I20260929 16:30:04.811254 17911 raft_consensus.cc:2781] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:30:04.811355 17911 raft_consensus.cc:482] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:30:04.811414 17911 raft_consensus.cc:3037] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:04.812500 17911 raft_consensus.cc:504] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5b0c1acfa54c58a63757838f2cdeb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "ea46ce661b18413bb4b81e3a4317643f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33915 } } peers { permanent_uuid: "5b1e8c6e1a5a4fc09ad341c53df60fce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45045 } }
I20260929 16:30:04.812888 17911 leader_election.cc:288] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f [CANDIDATE]: Term 1 election: Requested vote from peers 6c5b0c1acfa54c58a63757838f2cdeb6 (127.0.98.3:40829), 5b1e8c6e1a5a4fc09ad341c53df60fce (127.0.98.1:45045)
I20260929 16:30:04.813352 17574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f826a22f117e4742bc08965c6bfb0d16" candidate_uuid: "ea46ce661b18413bb4b81e3a4317643f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b1e8c6e1a5a4fc09ad341c53df60fce"
I20260929 16:30:04.813352 17842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f826a22f117e4742bc08965c6bfb0d16" candidate_uuid: "ea46ce661b18413bb4b81e3a4317643f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5b0c1acfa54c58a63757838f2cdeb6"
I20260929 16:30:04.813501 17842 raft_consensus.cc:3037] T f826a22f117e4742bc08965c6bfb0d16 P 6c5b0c1acfa54c58a63757838f2cdeb6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:04.813503 17574 raft_consensus.cc:3037] T f826a22f117e4742bc08965c6bfb0d16 P 5b1e8c6e1a5a4fc09ad341c53df60fce [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:04.814602 17842 raft_consensus.cc:2454] T f826a22f117e4742bc08965c6bfb0d16 P 6c5b0c1acfa54c58a63757838f2cdeb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea46ce661b18413bb4b81e3a4317643f in term 1.
I20260929 16:30:04.814910 17574 raft_consensus.cc:2454] T f826a22f117e4742bc08965c6bfb0d16 P 5b1e8c6e1a5a4fc09ad341c53df60fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea46ce661b18413bb4b81e3a4317643f in term 1.
I20260929 16:30:04.814972 17641 leader_election.cc:302] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c5b0c1acfa54c58a63757838f2cdeb6, ea46ce661b18413bb4b81e3a4317643f; no voters:
I20260929 16:30:04.815232 17911 raft_consensus.cc:2781] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:04.815495 17911 raft_consensus.cc:686] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f [term 1 LEADER]: Becoming Leader. State: Replica: ea46ce661b18413bb4b81e3a4317643f, State: Running, Role: LEADER
I20260929 16:30:04.815927 17911 consensus_queue.cc:237] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5b0c1acfa54c58a63757838f2cdeb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } } peers { permanent_uuid: "ea46ce661b18413bb4b81e3a4317643f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33915 } } peers { permanent_uuid: "5b1e8c6e1a5a4fc09ad341c53df60fce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45045 } }
I20260929 16:30:04.819667 17428 catalog_manager.cc:5495] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f reported cstate change: term changed from 0 to 1, leader changed from <none> to ea46ce661b18413bb4b81e3a4317643f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ea46ce661b18413bb4b81e3a4317643f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c5b0c1acfa54c58a63757838f2cdeb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea46ce661b18413bb4b81e3a4317643f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b1e8c6e1a5a4fc09ad341c53df60fce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45045 } health_report { overall_health: UNKNOWN } } }
W20260929 16:30:04.824851 17621 tablet.cc:2369] T f826a22f117e4742bc08965c6bfb0d16 P 5b1e8c6e1a5a4fc09ad341c53df60fce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:30:04.826503 17889 tablet.cc:2369] T f826a22f117e4742bc08965c6bfb0d16 P 6c5b0c1acfa54c58a63757838f2cdeb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:30:04.847903 17755 tablet.cc:2369] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:30:04.875133 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:04.871411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55254 (local address 127.0.98.2:33915)
0929 16:30:04.871573 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:04.871577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:04.871591 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:30:04.871695 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:04.871698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:04.871753 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:04.871855 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:30:04.872103 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.872603 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.873537 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:04.873756 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:04.873845 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:04.874166 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:30:04.874672 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:30:04.874770 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:04.874860 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 16:30:04.874912 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:30:04.881600 17842 raft_consensus.cc:1261] T f826a22f117e4742bc08965c6bfb0d16 P 6c5b0c1acfa54c58a63757838f2cdeb6 [term 1 FOLLOWER]: Refusing update from remote peer ea46ce661b18413bb4b81e3a4317643f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:04.881600 17574 raft_consensus.cc:1261] T f826a22f117e4742bc08965c6bfb0d16 P 5b1e8c6e1a5a4fc09ad341c53df60fce [term 1 FOLLOWER]: Refusing update from remote peer ea46ce661b18413bb4b81e3a4317643f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:04.882501 17915 consensus_queue.cc:1035] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c5b0c1acfa54c58a63757838f2cdeb6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:04.882736 17911 consensus_queue.cc:1035] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b1e8c6e1a5a4fc09ad341c53df60fce" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:04.890743 17918 mvcc.cc:205] Tried to move back new op lower bound from 7334704762386825216 to 7334704762129219584. Current Snapshot: MvccSnapshot[applied={T|T < 7334704762386825216}]
I20260929 16:30:04.891726 17919 mvcc.cc:205] Tried to move back new op lower bound from 7334704762386825216 to 7334704762129219584. Current Snapshot: MvccSnapshot[applied={T|T < 7334704762386825216}]
I20260929 16:30:04.893909 17920 mvcc.cc:205] Tried to move back new op lower bound from 7334704762386825216 to 7334704762129219584. Current Snapshot: MvccSnapshot[applied={T|T < 7334704762386825216 or (T in {7334704762386825216})}]
I20260929 16:30:04.899701 17429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:30:04.899931 17429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:30:04.902999 17429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e3f9025719243b690ec8645e92cec4c: Sending DeleteTablet for 3 replicas of tablet f826a22f117e4742bc08965c6bfb0d16
I20260929 16:30:04.904009 17688 tablet_service.cc:1514] Processing DeleteTablet for tablet f826a22f117e4742bc08965c6bfb0d16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55242
I20260929 16:30:04.904011 17822 tablet_service.cc:1514] Processing DeleteTablet for tablet f826a22f117e4742bc08965c6bfb0d16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43034
I20260929 16:30:04.904628 17928 tablet_replica.cc:330] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f: stopping tablet replica
I20260929 16:30:04.904620 17929 tablet_replica.cc:330] T f826a22f117e4742bc08965c6bfb0d16 P 6c5b0c1acfa54c58a63757838f2cdeb6: stopping tablet replica
I20260929 16:30:04.904601 17554 tablet_service.cc:1514] Processing DeleteTablet for tablet f826a22f117e4742bc08965c6bfb0d16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50632
I20260929 16:30:04.904978 17929 raft_consensus.cc:2229] T f826a22f117e4742bc08965c6bfb0d16 P 6c5b0c1acfa54c58a63757838f2cdeb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:30:04.905019 17928 raft_consensus.cc:2229] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:30:04.905408 17929 raft_consensus.cc:2258] T f826a22f117e4742bc08965c6bfb0d16 P 6c5b0c1acfa54c58a63757838f2cdeb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:04.905485 17928 raft_consensus.cc:2258] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:04.906018 17930 tablet_replica.cc:330] T f826a22f117e4742bc08965c6bfb0d16 P 5b1e8c6e1a5a4fc09ad341c53df60fce: stopping tablet replica
I20260929 16:30:04.906060 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 17491
I20260929 16:30:04.907712 17929 ts_tablet_manager.cc:1905] T f826a22f117e4742bc08965c6bfb0d16 P 6c5b0c1acfa54c58a63757838f2cdeb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:30:04.908404 17928 ts_tablet_manager.cc:1905] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:30:04.910665 17929 ts_tablet_manager.cc:1918] T f826a22f117e4742bc08965c6bfb0d16 P 6c5b0c1acfa54c58a63757838f2cdeb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:30:04.910813 17929 log.cc:1196] T f826a22f117e4742bc08965c6bfb0d16 P 6c5b0c1acfa54c58a63757838f2cdeb6: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/f826a22f117e4742bc08965c6bfb0d16
I20260929 16:30:04.911223 17929 ts_tablet_manager.cc:1939] T f826a22f117e4742bc08965c6bfb0d16 P 6c5b0c1acfa54c58a63757838f2cdeb6: Deleting consensus metadata
I20260929 16:30:04.911538 17928 ts_tablet_manager.cc:1918] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:30:04.911702 17928 log.cc:1196] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/f826a22f117e4742bc08965c6bfb0d16
I20260929 16:30:04.912073 17928 ts_tablet_manager.cc:1939] T f826a22f117e4742bc08965c6bfb0d16 P ea46ce661b18413bb4b81e3a4317643f: Deleting consensus metadata
I20260929 16:30:04.912753 17413 catalog_manager.cc:4841] TS 6c5b0c1acfa54c58a63757838f2cdeb6 (127.0.98.3:40829): tablet f826a22f117e4742bc08965c6bfb0d16 (table test-table [id=78fdb0807fa34f889f4670aad910679e]) successfully deleted
I20260929 16:30:04.913451 17414 catalog_manager.cc:4841] TS ea46ce661b18413bb4b81e3a4317643f (127.0.98.2:33915): tablet f826a22f117e4742bc08965c6bfb0d16 (table test-table [id=78fdb0807fa34f889f4670aad910679e]) successfully deleted
W20260929 16:30:04.913589 17415 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45045 (error 108)
I20260929 16:30:04.914660 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:04.914369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45045 (local address 127.0.0.1:50644)
0929 16:30:04.914491 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:04.914551 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 16:30:04.914799 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 17625
W20260929 16:30:04.914832 17415 catalog_manager.cc:4584] TS 5b1e8c6e1a5a4fc09ad341c53df60fce (127.0.98.1:45045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f826a22f117e4742bc08965c6bfb0d16: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45045: connect: Connection refused (error 111)
I20260929 16:30:04.920507 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 17759
I20260929 16:30:04.925662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 17397
2026-09-29T16:30:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2582 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T16:30:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:04Z Disabled control of system clock
I20260929 16:30:04.957800 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35819
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:30:05.072479 17935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:05.072793 17935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:05.072834 17935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:05.077310 17935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:05.077390 17935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:05.077405 17935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:05.077423 17935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:05.077440 17935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:05.082473 17935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35819
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:05.083710 17935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:05.085101 17935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:05.093819 17940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:05.093907 17935 server_base.cc:1034] running on GCE node
W20260929 16:30:05.093791 17941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:05.093823 17943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:05.094715 17935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:05.095952 17935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:05.097164 17935 hybrid_clock.cc:648] HybridClock initialized: now 1790699405097149 us; error 36 us; skew 500 ppm
I20260929 16:30:05.099211 17935 webserver.cc:460] Webserver started at http://127.0.98.62:46161/ using document root <none> and password file <none>
I20260929 16:30:05.099841 17935 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:05.099941 17935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:05.100167 17935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:05.102486 17935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "bf3d5ede5a46496d8aec661b9def1284"
format_stamp: "Formatted at 2026-09-29 16:30:05 on dist-test-slave-vq9v"
server_key: "13bb8cdc2eff125b872449d0acf42c89"
server_key_iv: "b9b8b7d454065bda11056e10205c3219"
server_key_version: "encryptionkey@0"
I20260929 16:30:05.103096 17935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "bf3d5ede5a46496d8aec661b9def1284"
format_stamp: "Formatted at 2026-09-29 16:30:05 on dist-test-slave-vq9v"
server_key: "13bb8cdc2eff125b872449d0acf42c89"
server_key_iv: "b9b8b7d454065bda11056e10205c3219"
server_key_version: "encryptionkey@0"
I20260929 16:30:05.107164 17935 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:30:05.109848 17949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:05.111198 17935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:05.111382 17935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "bf3d5ede5a46496d8aec661b9def1284"
format_stamp: "Formatted at 2026-09-29 16:30:05 on dist-test-slave-vq9v"
server_key: "13bb8cdc2eff125b872449d0acf42c89"
server_key_iv: "b9b8b7d454065bda11056e10205c3219"
server_key_version: "encryptionkey@0"
I20260929 16:30:05.111512 17935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:05.126586 17935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:05.127367 17935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:05.127588 17935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:05.135114 17935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35819
I20260929 16:30:05.135133 18001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35819 every 8 connection(s)
I20260929 16:30:05.136139 17935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:05.139515 18002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:05.144387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 17935
I20260929 16:30:05.144528 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:05.144806 392 external_mini_cluster.cc:1428] Setting key 3991a6f604d53871ad0e63fa86de06a3
I20260929 16:30:05.146193 18002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284: Bootstrap starting.
I20260929 16:30:05.148417 18002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:05.149287 18002 log.cc:824] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:05.151240 18002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284: No bootstrap required, opened a new log
I20260929 16:30:05.154983 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:05.146391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46036 (local address 127.0.98.62:35819)
0929 16:30:05.147174 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:05.147199 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:05.147252 (+ 53us) server_negotiation.cc:407] Connection header received
0929 16:30:05.147983 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:05.148019 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:05.148453 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:05.148896 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:05.149458 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.150438 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.151344 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.151852 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.152030 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.153003 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:05.153010 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:05.153024 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:05.153503 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:05.153565 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:05.153586 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:05.153709 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:05.153865 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 16:30:05.154158 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":153,"threads_started":1}
I20260929 16:30:05.155364 18002 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf3d5ede5a46496d8aec661b9def1284" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35819 } }
I20260929 16:30:05.155632 18002 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:05.155730 18002 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf3d5ede5a46496d8aec661b9def1284, State: Initialized, Role: FOLLOWER
I20260929 16:30:05.156240 18002 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf3d5ede5a46496d8aec661b9def1284" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35819 } }
I20260929 16:30:05.156402 18002 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:05.156493 18002 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:05.156594 18002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:05.157709 18002 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf3d5ede5a46496d8aec661b9def1284" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35819 } }
I20260929 16:30:05.158054 18002 leader_election.cc:302] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf3d5ede5a46496d8aec661b9def1284; no voters:
I20260929 16:30:05.158340 18002 leader_election.cc:288] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:05.158537 18007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:05.158785 18007 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284 [term 1 LEADER]: Becoming Leader. State: Replica: bf3d5ede5a46496d8aec661b9def1284, State: Running, Role: LEADER
I20260929 16:30:05.159094 18007 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf3d5ede5a46496d8aec661b9def1284" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35819 } }
I20260929 16:30:05.159617 18002 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:05.160703 18008 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf3d5ede5a46496d8aec661b9def1284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf3d5ede5a46496d8aec661b9def1284" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35819 } } }
I20260929 16:30:05.160950 18008 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:05.161659 18009 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf3d5ede5a46496d8aec661b9def1284. Latest consensus state: current_term: 1 leader_uuid: "bf3d5ede5a46496d8aec661b9def1284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf3d5ede5a46496d8aec661b9def1284" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35819 } } }
I20260929 16:30:05.161777 18009 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284 [sys.catalog]: This master's current role is: LEADER
W20260929 16:30:05.164937 18022 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:30:05.165032 18022 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:30:05.165156 18020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:05.166285 18020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:05.172197 18020 catalog_manager.cc:1347] Generated new cluster ID: 5283adbaefd846ec922fe93a220dbb80
I20260929 16:30:05.172291 18020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:05.189134 18020 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:05.190502 18020 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:05.197471 18020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284: Generated new TSK 0
I20260929 16:30:05.198407 18020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:05.213693 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35819
--builtin_ntp_servers=127.0.98.20:42213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:30:05.330443 18026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:05.330744 18026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:05.330783 18026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:05.334962 18026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:05.335033 18026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:05.335090 18026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:30:05.339998 18026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:05.341226 18026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:05.342734 18026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:05.352893 18031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:05.352905 18034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:05.352921 18032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:05.353250 18026 server_base.cc:1034] running on GCE node
I20260929 16:30:05.353793 18026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:05.354413 18026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:05.355583 18026 hybrid_clock.cc:648] HybridClock initialized: now 1790699405355565 us; error 72 us; skew 500 ppm
I20260929 16:30:05.357709 18026 webserver.cc:460] Webserver started at http://127.0.98.1:39239/ using document root <none> and password file <none>
I20260929 16:30:05.358259 18026 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:05.358321 18026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:05.358489 18026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:05.360775 18026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "7a6ec3a3205b4b1d87c68ef59f30a81a"
format_stamp: "Formatted at 2026-09-29 16:30:05 on dist-test-slave-vq9v"
server_key: "a27c850496cb81063975a05da351500d"
server_key_iv: "4cd8d541c421ca603bd9fe27fcdb5899"
server_key_version: "encryptionkey@0"
I20260929 16:30:05.361443 18026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7a6ec3a3205b4b1d87c68ef59f30a81a"
format_stamp: "Formatted at 2026-09-29 16:30:05 on dist-test-slave-vq9v"
server_key: "a27c850496cb81063975a05da351500d"
server_key_iv: "4cd8d541c421ca603bd9fe27fcdb5899"
server_key_version: "encryptionkey@0"
I20260929 16:30:05.365502 18026 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:30:05.368180 18040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:05.369822 18026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:05.369987 18026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "7a6ec3a3205b4b1d87c68ef59f30a81a"
format_stamp: "Formatted at 2026-09-29 16:30:05 on dist-test-slave-vq9v"
server_key: "a27c850496cb81063975a05da351500d"
server_key_iv: "4cd8d541c421ca603bd9fe27fcdb5899"
server_key_version: "encryptionkey@0"
I20260929 16:30:05.370126 18026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:05.383534 18026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:05.384358 18026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:05.384594 18026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:05.385298 18026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:05.386317 18026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:05.386394 18026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:05.386467 18026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:05.386520 18026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:05.396070 18026 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35103
I20260929 16:30:05.396095 18153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35103 every 8 connection(s)
I20260929 16:30:05.397053 18026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:30:05.401280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 18026
I20260929 16:30:05.401418 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:30:05.401805 392 external_mini_cluster.cc:1428] Setting key 8856af2ebce1ab2c135f8a77897b7a27
I20260929 16:30:05.404037 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35819
--builtin_ntp_servers=127.0.98.20:42213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:30:05.408407 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:05.399152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58735 (local address 127.0.98.62:35819)
0929 16:30:05.399296 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:05.399300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:05.400506 (+ 1206us) server_negotiation.cc:407] Connection header received
0929 16:30:05.402447 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:05.402450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:05.402529 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:05.402641 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:05.403767 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.404428 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.406093 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.406340 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.406461 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.407694 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:05.407699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:05.407707 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:05.407721 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:05.407742 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:05.407782 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:05.407866 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:05.408096 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 16:30:05.408187 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:30:05.409324 18156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:05.399617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35819 (local address 127.0.98.1:58735)
0929 16:30:05.400320 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:05.400381 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:05.402155 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:05.402844 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:05.402852 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:05.402946 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:05.403581 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:05.403604 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.404577 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.404584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:05.405941 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:05.405951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.406514 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.406518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:05.406704 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.407390 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:30:05.407391 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:05.407437 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:30:05.407513 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:05.407914 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:05.407923 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:05.407933 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 16:30:05.408043 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 16:30:05.408312 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":120,"threads_started":1}
I20260929 16:30:05.410765 18154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35819
I20260929 16:30:05.411036 18154 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:05.411635 18154 heartbeater.cc:511] Master 127.0.98.62:35819 requested a full tablet report, sending...
I20260929 16:30:05.413374 17966 ts_manager.cc:194] Registered new tserver with Master: 7a6ec3a3205b4b1d87c68ef59f30a81a (127.0.98.1:35103)
I20260929 16:30:05.414572 17966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58735
W20260929 16:30:05.521860 18157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:05.522140 18157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:05.522181 18157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:05.526330 18157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:05.526396 18157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:05.526455 18157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:30:05.531353 18157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:05.532450 18157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:05.533878 18157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:05.543917 18163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:05.543879 18165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:05.543900 18157 server_base.cc:1034] running on GCE node
W20260929 16:30:05.543924 18162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:05.544618 18157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:05.545439 18157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:05.546656 18157 hybrid_clock.cc:648] HybridClock initialized: now 1790699405546620 us; error 50 us; skew 500 ppm
I20260929 16:30:05.548751 18157 webserver.cc:460] Webserver started at http://127.0.98.2:39701/ using document root <none> and password file <none>
I20260929 16:30:05.549464 18157 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:05.549537 18157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:05.549717 18157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:05.551796 18157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "3339b57741e044b2aed58b42a5b43623"
format_stamp: "Formatted at 2026-09-29 16:30:05 on dist-test-slave-vq9v"
server_key: "f5e0e733678a1a05f7c6f89c7dcd8b37"
server_key_iv: "ff5c8a3625d3ff6c71f755adf833614c"
server_key_version: "encryptionkey@0"
I20260929 16:30:05.552326 18157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3339b57741e044b2aed58b42a5b43623"
format_stamp: "Formatted at 2026-09-29 16:30:05 on dist-test-slave-vq9v"
server_key: "f5e0e733678a1a05f7c6f89c7dcd8b37"
server_key_iv: "ff5c8a3625d3ff6c71f755adf833614c"
server_key_version: "encryptionkey@0"
I20260929 16:30:05.556082 18157 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:30:05.558630 18171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:05.560065 18157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:30:05.560205 18157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "3339b57741e044b2aed58b42a5b43623"
format_stamp: "Formatted at 2026-09-29 16:30:05 on dist-test-slave-vq9v"
server_key: "f5e0e733678a1a05f7c6f89c7dcd8b37"
server_key_iv: "ff5c8a3625d3ff6c71f755adf833614c"
server_key_version: "encryptionkey@0"
I20260929 16:30:05.560320 18157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:05.579007 18157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:05.579831 18157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:05.580087 18157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:05.580736 18157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:05.581827 18157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:05.581904 18157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:05.581979 18157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:05.582024 18157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:05.591218 18157 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40079
I20260929 16:30:05.591243 18284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40079 every 8 connection(s)
I20260929 16:30:05.592178 18157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:30:05.601445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 18157
I20260929 16:30:05.601578 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:30:05.601567 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:05.594148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32981 (local address 127.0.98.62:35819)
0929 16:30:05.594340 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:05.594345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:05.595429 (+ 1084us) server_negotiation.cc:407] Connection header received
0929 16:30:05.596809 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:05.596811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:05.596860 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:05.596937 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:05.597868 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.598404 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.599512 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.599717 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.599802 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.600939 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:05.600941 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:05.600943 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:05.600955 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:05.600969 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:05.600976 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:05.601033 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:05.601337 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 16:30:05.601408 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:30:05.601926 392 external_mini_cluster.cc:1428] Setting key dfcacd194da0302fddecd2b657e7a11d
I20260929 16:30:05.602698 18287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:05.594601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35819 (local address 127.0.98.2:32981)
0929 16:30:05.595273 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:05.595313 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:05.596612 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:05.597100 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:05.597107 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:05.597190 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:05.597732 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:05.597746 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.598514 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.598517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:05.599401 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:05.599410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.599853 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.599856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:05.599993 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.600670 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:30:05.600672 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:05.600716 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:30:05.600784 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:05.601076 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:05.601119 (+ 43us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:05.601129 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 16:30:05.601326 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 16:30:05.601663 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":136,"threads_started":1}
I20260929 16:30:05.603977 18285 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35819
I20260929 16:30:05.604245 18285 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:05.604240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35819
--builtin_ntp_servers=127.0.98.20:42213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:30:05.604821 18285 heartbeater.cc:511] Master 127.0.98.62:35819 requested a full tablet report, sending...
I20260929 16:30:05.606045 17966 ts_manager.cc:194] Registered new tserver with Master: 3339b57741e044b2aed58b42a5b43623 (127.0.98.2:40079)
I20260929 16:30:05.606698 17966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:32981
W20260929 16:30:05.723479 18288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:05.723826 18288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:05.723902 18288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:05.728143 18288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:05.728259 18288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:05.728360 18288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:30:05.733652 18288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:05.734889 18288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:05.736377 18288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:05.746925 18294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:05.747114 18288 server_base.cc:1034] running on GCE node
W20260929 16:30:05.747350 18296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:05.747413 18293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:05.747942 18288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:05.748684 18288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:05.749902 18288 hybrid_clock.cc:648] HybridClock initialized: now 1790699405749878 us; error 41 us; skew 500 ppm
I20260929 16:30:05.752120 18288 webserver.cc:460] Webserver started at http://127.0.98.3:39535/ using document root <none> and password file <none>
I20260929 16:30:05.752817 18288 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:05.752925 18288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:05.753158 18288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:05.755463 18288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "3ff55a1ba2bd49de89438c23a80b4ce1"
format_stamp: "Formatted at 2026-09-29 16:30:05 on dist-test-slave-vq9v"
server_key: "01277e5024a5c881b6df830c565670a8"
server_key_iv: "6cf1f39cfdcb9989211e043941bffc0c"
server_key_version: "encryptionkey@0"
I20260929 16:30:05.756084 18288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3ff55a1ba2bd49de89438c23a80b4ce1"
format_stamp: "Formatted at 2026-09-29 16:30:05 on dist-test-slave-vq9v"
server_key: "01277e5024a5c881b6df830c565670a8"
server_key_iv: "6cf1f39cfdcb9989211e043941bffc0c"
server_key_version: "encryptionkey@0"
I20260929 16:30:05.760298 18288 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:30:05.763391 18302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:05.764936 18288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:30:05.765072 18288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "3ff55a1ba2bd49de89438c23a80b4ce1"
format_stamp: "Formatted at 2026-09-29 16:30:05 on dist-test-slave-vq9v"
server_key: "01277e5024a5c881b6df830c565670a8"
server_key_iv: "6cf1f39cfdcb9989211e043941bffc0c"
server_key_version: "encryptionkey@0"
I20260929 16:30:05.765233 18288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:05.789563 18288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:05.790290 18288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:05.790500 18288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:05.791136 18288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:05.792095 18288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:05.792176 18288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:05.792268 18288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:05.792320 18288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:05.802598 18288 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45067
I20260929 16:30:05.802682 18415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45067 every 8 connection(s)
I20260929 16:30:05.803596 18288 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:30:05.812057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 18288
I20260929 16:30:05.812217 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:30:05.812541 392 external_mini_cluster.cc:1428] Setting key 2b0d547a0e8fe2ab9cf5a9267c7c5a82
I20260929 16:30:05.813978 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:05.805686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54479 (local address 127.0.98.62:35819)
0929 16:30:05.805877 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:05.805883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:05.807213 (+ 1330us) server_negotiation.cc:407] Connection header received
0929 16:30:05.808790 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:05.808794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:05.808842 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:05.808928 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:05.810041 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.810568 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.811786 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.812001 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.812118 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.813420 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:05.813424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:05.813427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:05.813440 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:05.813462 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:05.813472 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:05.813568 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:05.813744 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 16:30:05.813817 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:30:05.815369 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:05.806192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35819 (local address 127.0.98.3:54479)
0929 16:30:05.807007 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:05.807064 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:05.808559 (+ 1495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:05.809127 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:05.809135 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:05.809306 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:05.809887 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:05.809903 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.810689 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.810692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:05.811673 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:05.811683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.812176 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.812180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:05.812397 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.813080 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:30:05.813081 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:05.813143 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:30:05.813275 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:05.813576 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:05.813588 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:05.813594 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:30:05.813740 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 16:30:05.814183 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":196,"threads_started":1}
I20260929 16:30:05.816727 18416 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35819
I20260929 16:30:05.817004 18416 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:05.817644 18416 heartbeater.cc:511] Master 127.0.98.62:35819 requested a full tablet report, sending...
I20260929 16:30:05.818851 17966 ts_manager.cc:194] Registered new tserver with Master: 3ff55a1ba2bd49de89438c23a80b4ce1 (127.0.98.3:45067)
I20260929 16:30:05.819455 17966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:54479
I20260929 16:30:05.827261 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:30:05.833415 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:05.829348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46046 (local address 127.0.98.62:35819)
0929 16:30:05.829494 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:05.829499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:05.829703 (+ 204us) server_negotiation.cc:407] Connection header received
0929 16:30:05.829873 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:05.829875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:05.829926 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:05.830010 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:05.830276 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.830830 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.831748 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.831965 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.832047 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.832847 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:05.832850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:05.832852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:05.832864 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:05.832882 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:05.832890 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:05.832947 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:05.833080 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 16:30:05.833133 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:30:05.836565 17966 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:30:05.839898 17966 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:30:05.855384 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:05.849706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45067 (local address 127.0.0.1:50448)
0929 16:30:05.850285 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:05.850300 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:05.850374 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:05.851087 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:05.851091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:05.851110 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:05.851197 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:05.851207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.852507 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.852514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:05.853678 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:05.853687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.854702 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.854710 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:05.854825 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.854866 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 16:30:05.854990 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 16:30:05.855184 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":42,"thread_start_us":97,"threads_started":1}
I20260929 16:30:05.855623 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:05.848940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44614 (local address 127.0.98.1:35103)
0929 16:30:05.849794 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:05.849808 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:05.849835 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:30:05.849915 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:05.849921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:05.850163 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:05.850814 (+ 651us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:05.850957 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.852178 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.854040 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.854815 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.855116 (+ 301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.855311 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:05.855362 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 16:30:05.855467 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":131,"threads_started":1}
I20260929 16:30:05.855731 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:05.849143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49790 (local address 127.0.98.2:40079)
0929 16:30:05.850095 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:05.850106 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:05.850130 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:30:05.850344 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:05.850351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:05.850553 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:05.850731 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:05.851149 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.851963 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.854080 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.854672 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.854787 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.854982 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:05.855453 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 16:30:05.855549 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":855,"thread_start_us":180,"threads_started":1}
I20260929 16:30:05.855572 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:05.848798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35103 (local address 127.0.0.1:44614)
0929 16:30:05.849467 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:05.849521 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:05.849652 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:05.850358 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:05.850364 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:05.850383 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:05.850543 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:05.850551 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.852312 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.852316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:05.853901 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:05.853913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.854956 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.854960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:05.855186 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.855212 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 16:30:05.855284 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 16:30:05.855349 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":189,"thread_start_us":113,"threads_started":1}
I20260929 16:30:05.855656 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:05.848941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40079 (local address 127.0.0.1:49790)
0929 16:30:05.850031 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:05.850046 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:05.850152 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:05.850893 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:05.850895 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:05.850911 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:05.850985 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:05.850990 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.852098 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.852101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:05.853901 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:05.853913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.854956 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.854960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:05.855129 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.855151 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:30:05.855239 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 16:30:05.855312 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1026,"mutex_wait_us":268,"thread_start_us":121,"threads_started":1}
I20260929 16:30:05.856482 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:05.850026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.0.98.3:45067)
0929 16:30:05.850624 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:05.850636 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:05.850658 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:30:05.850743 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:05.850750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:05.850929 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:05.851076 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:05.851334 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.852397 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.853800 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.854558 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.854674 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.854892 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:05.856187 (+ 1295us) server_negotiation.cc:299] Negotiation successful
0929 16:30:05.856302 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":157,"threads_started":1}
I20260929 16:30:05.859014 18088 tablet_service.cc:1467] Processing CreateTablet for tablet b770ca6b62834c63872a26b7ae92eb7b (DEFAULT_TABLE table=test-table [id=94cf3dbff26f410ab54c974da2cecbbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:05.859277 18350 tablet_service.cc:1467] Processing CreateTablet for tablet b770ca6b62834c63872a26b7ae92eb7b (DEFAULT_TABLE table=test-table [id=94cf3dbff26f410ab54c974da2cecbbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:05.859120 18219 tablet_service.cc:1467] Processing CreateTablet for tablet b770ca6b62834c63872a26b7ae92eb7b (DEFAULT_TABLE table=test-table [id=94cf3dbff26f410ab54c974da2cecbbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:05.860064 18088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b770ca6b62834c63872a26b7ae92eb7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:05.860145 18350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b770ca6b62834c63872a26b7ae92eb7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:05.860190 18219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b770ca6b62834c63872a26b7ae92eb7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:05.867430 18433 tablet_bootstrap.cc:492] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a: Bootstrap starting.
I20260929 16:30:05.868672 18434 tablet_bootstrap.cc:492] T b770ca6b62834c63872a26b7ae92eb7b P 3339b57741e044b2aed58b42a5b43623: Bootstrap starting.
I20260929 16:30:05.869516 18435 tablet_bootstrap.cc:492] T b770ca6b62834c63872a26b7ae92eb7b P 3ff55a1ba2bd49de89438c23a80b4ce1: Bootstrap starting.
I20260929 16:30:05.870088 18433 tablet_bootstrap.cc:654] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:05.870873 18433 log.cc:824] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:05.871016 18434 tablet_bootstrap.cc:654] T b770ca6b62834c63872a26b7ae92eb7b P 3339b57741e044b2aed58b42a5b43623: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:05.871537 18435 tablet_bootstrap.cc:654] T b770ca6b62834c63872a26b7ae92eb7b P 3ff55a1ba2bd49de89438c23a80b4ce1: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:05.872150 18434 log.cc:824] T b770ca6b62834c63872a26b7ae92eb7b P 3339b57741e044b2aed58b42a5b43623: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:05.872327 18435 log.cc:824] T b770ca6b62834c63872a26b7ae92eb7b P 3ff55a1ba2bd49de89438c23a80b4ce1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:05.874490 18433 tablet_bootstrap.cc:492] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a: No bootstrap required, opened a new log
I20260929 16:30:05.874490 18434 tablet_bootstrap.cc:492] T b770ca6b62834c63872a26b7ae92eb7b P 3339b57741e044b2aed58b42a5b43623: No bootstrap required, opened a new log
I20260929 16:30:05.874709 18434 ts_tablet_manager.cc:1397] T b770ca6b62834c63872a26b7ae92eb7b P 3339b57741e044b2aed58b42a5b43623: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:30:05.874718 18433 ts_tablet_manager.cc:1397] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 16:30:05.879364 18434 raft_consensus.cc:348] T b770ca6b62834c63872a26b7ae92eb7b P 3339b57741e044b2aed58b42a5b43623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6ec3a3205b4b1d87c68ef59f30a81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35103 } } peers { permanent_uuid: "3ff55a1ba2bd49de89438c23a80b4ce1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45067 } } peers { permanent_uuid: "3339b57741e044b2aed58b42a5b43623" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40079 } }
I20260929 16:30:05.879364 18433 raft_consensus.cc:348] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6ec3a3205b4b1d87c68ef59f30a81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35103 } } peers { permanent_uuid: "3ff55a1ba2bd49de89438c23a80b4ce1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45067 } } peers { permanent_uuid: "3339b57741e044b2aed58b42a5b43623" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40079 } }
I20260929 16:30:05.879675 18434 raft_consensus.cc:374] T b770ca6b62834c63872a26b7ae92eb7b P 3339b57741e044b2aed58b42a5b43623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:05.879674 18433 raft_consensus.cc:374] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:05.879729 18434 raft_consensus.cc:729] T b770ca6b62834c63872a26b7ae92eb7b P 3339b57741e044b2aed58b42a5b43623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3339b57741e044b2aed58b42a5b43623, State: Initialized, Role: FOLLOWER
I20260929 16:30:05.879745 18433 raft_consensus.cc:729] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6ec3a3205b4b1d87c68ef59f30a81a, State: Initialized, Role: FOLLOWER
I20260929 16:30:05.879876 18435 tablet_bootstrap.cc:492] T b770ca6b62834c63872a26b7ae92eb7b P 3ff55a1ba2bd49de89438c23a80b4ce1: No bootstrap required, opened a new log
I20260929 16:30:05.880129 18435 ts_tablet_manager.cc:1397] T b770ca6b62834c63872a26b7ae92eb7b P 3ff55a1ba2bd49de89438c23a80b4ce1: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260929 16:30:05.880246 18433 consensus_queue.cc:260] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6ec3a3205b4b1d87c68ef59f30a81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35103 } } peers { permanent_uuid: "3ff55a1ba2bd49de89438c23a80b4ce1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45067 } } peers { permanent_uuid: "3339b57741e044b2aed58b42a5b43623" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40079 } }
I20260929 16:30:05.880246 18434 consensus_queue.cc:260] T b770ca6b62834c63872a26b7ae92eb7b P 3339b57741e044b2aed58b42a5b43623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6ec3a3205b4b1d87c68ef59f30a81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35103 } } peers { permanent_uuid: "3ff55a1ba2bd49de89438c23a80b4ce1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45067 } } peers { permanent_uuid: "3339b57741e044b2aed58b42a5b43623" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40079 } }
I20260929 16:30:05.881188 18154 heartbeater.cc:503] Master 127.0.98.62:35819 was elected leader, sending a full tablet report...
I20260929 16:30:05.881343 18433 ts_tablet_manager.cc:1428] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260929 16:30:05.881343 18434 ts_tablet_manager.cc:1428] T b770ca6b62834c63872a26b7ae92eb7b P 3339b57741e044b2aed58b42a5b43623: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260929 16:30:05.882400 18285 heartbeater.cc:503] Master 127.0.98.62:35819 was elected leader, sending a full tablet report...
I20260929 16:30:05.884637 18435 raft_consensus.cc:348] T b770ca6b62834c63872a26b7ae92eb7b P 3ff55a1ba2bd49de89438c23a80b4ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6ec3a3205b4b1d87c68ef59f30a81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35103 } } peers { permanent_uuid: "3ff55a1ba2bd49de89438c23a80b4ce1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45067 } } peers { permanent_uuid: "3339b57741e044b2aed58b42a5b43623" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40079 } }
I20260929 16:30:05.884966 18435 raft_consensus.cc:374] T b770ca6b62834c63872a26b7ae92eb7b P 3ff55a1ba2bd49de89438c23a80b4ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:05.885037 18435 raft_consensus.cc:729] T b770ca6b62834c63872a26b7ae92eb7b P 3ff55a1ba2bd49de89438c23a80b4ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ff55a1ba2bd49de89438c23a80b4ce1, State: Initialized, Role: FOLLOWER
I20260929 16:30:05.885676 18435 consensus_queue.cc:260] T b770ca6b62834c63872a26b7ae92eb7b P 3ff55a1ba2bd49de89438c23a80b4ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6ec3a3205b4b1d87c68ef59f30a81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35103 } } peers { permanent_uuid: "3ff55a1ba2bd49de89438c23a80b4ce1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45067 } } peers { permanent_uuid: "3339b57741e044b2aed58b42a5b43623" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40079 } }
I20260929 16:30:05.886415 18416 heartbeater.cc:503] Master 127.0.98.62:35819 was elected leader, sending a full tablet report...
I20260929 16:30:05.886862 18435 ts_tablet_manager.cc:1428] T b770ca6b62834c63872a26b7ae92eb7b P 3ff55a1ba2bd49de89438c23a80b4ce1: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260929 16:30:05.898458 18155 tablet.cc:2369] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:30:05.960615 18440 raft_consensus.cc:482] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:30:05.960902 18440 raft_consensus.cc:504] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6ec3a3205b4b1d87c68ef59f30a81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35103 } } peers { permanent_uuid: "3ff55a1ba2bd49de89438c23a80b4ce1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45067 } } peers { permanent_uuid: "3339b57741e044b2aed58b42a5b43623" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40079 } }
I20260929 16:30:05.962392 18440 leader_election.cc:288] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ff55a1ba2bd49de89438c23a80b4ce1 (127.0.98.3:45067), 3339b57741e044b2aed58b42a5b43623 (127.0.98.2:40079)
I20260929 16:30:05.966708 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:05.962816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50681 (local address 127.0.98.2:40079)
0929 16:30:05.962960 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:05.962965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:05.963155 (+ 190us) server_negotiation.cc:407] Connection header received
0929 16:30:05.963326 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:05.963329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:05.963383 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:05.963470 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:05.963778 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.964371 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.965732 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.966130 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.966259 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.966321 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:05.966500 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 16:30:05.966569 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:30:05.966727 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:05.962699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40079 (local address 127.0.98.1:50681)
0929 16:30:05.963059 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:05.963070 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:05.963209 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:05.963482 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:05.963485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:05.963501 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:05.963633 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:05.963638 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.964481 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.964486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:05.965606 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:05.965616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.966265 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.966269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:05.966363 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.966386 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:30:05.966484 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 16:30:05.966557 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":50,"thread_start_us":114,"threads_started":1}
I20260929 16:30:05.966997 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:05.962749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57277 (local address 127.0.98.3:45067)
0929 16:30:05.962935 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:05.962942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:05.963061 (+ 119us) server_negotiation.cc:407] Connection header received
0929 16:30:05.963259 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:05.963263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:05.963317 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:05.963411 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:05.963730 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.964503 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.965942 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.966390 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.966525 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.966587 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:05.966800 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 16:30:05.966880 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 16:30:05.967018 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:05.962600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45067 (local address 127.0.98.1:57277)
0929 16:30:05.962934 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:05.962952 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:05.963128 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:05.963436 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:05.963440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:05.963458 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:05.963603 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:05.963610 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.964658 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.964664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:05.965801 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:05.965811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:05.966535 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:05.966539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:05.966650 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:05.966672 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:30:05.966783 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 16:30:05.966859 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":28,"thread_start_us":113,"threads_started":1}
I20260929 16:30:05.967437 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b770ca6b62834c63872a26b7ae92eb7b" candidate_uuid: "7a6ec3a3205b4b1d87c68ef59f30a81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3339b57741e044b2aed58b42a5b43623" is_pre_election: true
I20260929 16:30:05.967626 18370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b770ca6b62834c63872a26b7ae92eb7b" candidate_uuid: "7a6ec3a3205b4b1d87c68ef59f30a81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ff55a1ba2bd49de89438c23a80b4ce1" is_pre_election: true
I20260929 16:30:05.967767 18239 raft_consensus.cc:2454] T b770ca6b62834c63872a26b7ae92eb7b P 3339b57741e044b2aed58b42a5b43623 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a6ec3a3205b4b1d87c68ef59f30a81a in term 0.
I20260929 16:30:05.967974 18370 raft_consensus.cc:2454] T b770ca6b62834c63872a26b7ae92eb7b P 3ff55a1ba2bd49de89438c23a80b4ce1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a6ec3a3205b4b1d87c68ef59f30a81a in term 0.
I20260929 16:30:05.968320 18041 leader_election.cc:302] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3339b57741e044b2aed58b42a5b43623, 7a6ec3a3205b4b1d87c68ef59f30a81a; no voters:
I20260929 16:30:05.968683 18440 raft_consensus.cc:2781] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:30:05.968781 18440 raft_consensus.cc:482] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:30:05.968815 18440 raft_consensus.cc:3037] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:05.970082 18440 raft_consensus.cc:504] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6ec3a3205b4b1d87c68ef59f30a81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35103 } } peers { permanent_uuid: "3ff55a1ba2bd49de89438c23a80b4ce1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45067 } } peers { permanent_uuid: "3339b57741e044b2aed58b42a5b43623" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40079 } }
I20260929 16:30:05.970520 18440 leader_election.cc:288] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a [CANDIDATE]: Term 1 election: Requested vote from peers 3ff55a1ba2bd49de89438c23a80b4ce1 (127.0.98.3:45067), 3339b57741e044b2aed58b42a5b43623 (127.0.98.2:40079)
I20260929 16:30:05.970958 18239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b770ca6b62834c63872a26b7ae92eb7b" candidate_uuid: "7a6ec3a3205b4b1d87c68ef59f30a81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3339b57741e044b2aed58b42a5b43623"
I20260929 16:30:05.970947 18370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b770ca6b62834c63872a26b7ae92eb7b" candidate_uuid: "7a6ec3a3205b4b1d87c68ef59f30a81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ff55a1ba2bd49de89438c23a80b4ce1"
I20260929 16:30:05.971119 18370 raft_consensus.cc:3037] T b770ca6b62834c63872a26b7ae92eb7b P 3ff55a1ba2bd49de89438c23a80b4ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:05.971123 18239 raft_consensus.cc:3037] T b770ca6b62834c63872a26b7ae92eb7b P 3339b57741e044b2aed58b42a5b43623 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:05.972255 18239 raft_consensus.cc:2454] T b770ca6b62834c63872a26b7ae92eb7b P 3339b57741e044b2aed58b42a5b43623 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a6ec3a3205b4b1d87c68ef59f30a81a in term 1.
I20260929 16:30:05.972290 18370 raft_consensus.cc:2454] T b770ca6b62834c63872a26b7ae92eb7b P 3ff55a1ba2bd49de89438c23a80b4ce1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a6ec3a3205b4b1d87c68ef59f30a81a in term 1.
I20260929 16:30:05.972602 18043 leader_election.cc:302] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ff55a1ba2bd49de89438c23a80b4ce1, 7a6ec3a3205b4b1d87c68ef59f30a81a; no voters:
I20260929 16:30:05.972811 18440 raft_consensus.cc:2781] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:05.973065 18440 raft_consensus.cc:686] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a [term 1 LEADER]: Becoming Leader. State: Replica: 7a6ec3a3205b4b1d87c68ef59f30a81a, State: Running, Role: LEADER
I20260929 16:30:05.973470 18440 consensus_queue.cc:237] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6ec3a3205b4b1d87c68ef59f30a81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35103 } } peers { permanent_uuid: "3ff55a1ba2bd49de89438c23a80b4ce1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45067 } } peers { permanent_uuid: "3339b57741e044b2aed58b42a5b43623" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40079 } }
I20260929 16:30:05.976723 17965 catalog_manager.cc:5495] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a6ec3a3205b4b1d87c68ef59f30a81a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7a6ec3a3205b4b1d87c68ef59f30a81a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a6ec3a3205b4b1d87c68ef59f30a81a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ff55a1ba2bd49de89438c23a80b4ce1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3339b57741e044b2aed58b42a5b43623" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40079 } health_report { overall_health: UNKNOWN } } }
I20260929 16:30:06.010370 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:06.006814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44624 (local address 127.0.98.1:35103)
0929 16:30:06.006946 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:06.006950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:06.006965 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:30:06.007041 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:06.007044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:06.007100 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:06.007225 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:30:06.007523 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:06.008043 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:06.008881 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:06.009142 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:06.009339 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:06.009594 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:30:06.010038 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:30:06.010134 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:06.010204 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 16:30:06.010255 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:30:06.016702 18239 raft_consensus.cc:1261] T b770ca6b62834c63872a26b7ae92eb7b P 3339b57741e044b2aed58b42a5b43623 [term 1 FOLLOWER]: Refusing update from remote peer 7a6ec3a3205b4b1d87c68ef59f30a81a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:06.016705 18370 raft_consensus.cc:1261] T b770ca6b62834c63872a26b7ae92eb7b P 3ff55a1ba2bd49de89438c23a80b4ce1 [term 1 FOLLOWER]: Refusing update from remote peer 7a6ec3a3205b4b1d87c68ef59f30a81a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:06.017506 18444 consensus_queue.cc:1035] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ff55a1ba2bd49de89438c23a80b4ce1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:06.017720 18440 consensus_queue.cc:1035] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3339b57741e044b2aed58b42a5b43623" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:06.027129 18448 mvcc.cc:205] Tried to move back new op lower bound from 7334704767036538880 to 7334704766870110208. Current Snapshot: MvccSnapshot[applied={T|T < 7334704767036538880}]
I20260929 16:30:06.035379 17965 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:30:06.035607 17965 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:30:06.038229 17965 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf3d5ede5a46496d8aec661b9def1284: Sending DeleteTablet for 3 replicas of tablet b770ca6b62834c63872a26b7ae92eb7b
I20260929 16:30:06.039196 18219 tablet_service.cc:1514] Processing DeleteTablet for tablet b770ca6b62834c63872a26b7ae92eb7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:06 UTC) from {username='slave'} at 127.0.0.1:49790
I20260929 16:30:06.039273 18350 tablet_service.cc:1514] Processing DeleteTablet for tablet b770ca6b62834c63872a26b7ae92eb7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:06 UTC) from {username='slave'} at 127.0.0.1:50448
I20260929 16:30:06.039394 18088 tablet_service.cc:1514] Processing DeleteTablet for tablet b770ca6b62834c63872a26b7ae92eb7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:06 UTC) from {username='slave'} at 127.0.0.1:44614
I20260929 16:30:06.039767 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 18026
I20260929 16:30:06.039978 18459 tablet_replica.cc:330] T b770ca6b62834c63872a26b7ae92eb7b P 7a6ec3a3205b4b1d87c68ef59f30a81a: stopping tablet replica
I20260929 16:30:06.040416 18457 tablet_replica.cc:330] T b770ca6b62834c63872a26b7ae92eb7b P 3339b57741e044b2aed58b42a5b43623: stopping tablet replica
I20260929 16:30:06.040695 18457 raft_consensus.cc:2229] T b770ca6b62834c63872a26b7ae92eb7b P 3339b57741e044b2aed58b42a5b43623 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:30:06.041030 18457 raft_consensus.cc:2258] T b770ca6b62834c63872a26b7ae92eb7b P 3339b57741e044b2aed58b42a5b43623 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:06.041378 18458 tablet_replica.cc:330] T b770ca6b62834c63872a26b7ae92eb7b P 3ff55a1ba2bd49de89438c23a80b4ce1: stopping tablet replica
I20260929 16:30:06.041697 18458 raft_consensus.cc:2229] T b770ca6b62834c63872a26b7ae92eb7b P 3ff55a1ba2bd49de89438c23a80b4ce1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:30:06.042006 18458 raft_consensus.cc:2258] T b770ca6b62834c63872a26b7ae92eb7b P 3ff55a1ba2bd49de89438c23a80b4ce1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:06.042964 18457 ts_tablet_manager.cc:1905] T b770ca6b62834c63872a26b7ae92eb7b P 3339b57741e044b2aed58b42a5b43623: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:30:06.043473 18458 ts_tablet_manager.cc:1905] T b770ca6b62834c63872a26b7ae92eb7b P 3ff55a1ba2bd49de89438c23a80b4ce1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:30:06.045872 18458 ts_tablet_manager.cc:1918] T b770ca6b62834c63872a26b7ae92eb7b P 3ff55a1ba2bd49de89438c23a80b4ce1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:30:06.046003 18458 log.cc:1196] T b770ca6b62834c63872a26b7ae92eb7b P 3ff55a1ba2bd49de89438c23a80b4ce1: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/b770ca6b62834c63872a26b7ae92eb7b
I20260929 16:30:06.046419 18458 ts_tablet_manager.cc:1939] T b770ca6b62834c63872a26b7ae92eb7b P 3ff55a1ba2bd49de89438c23a80b4ce1: Deleting consensus metadata
I20260929 16:30:06.046626 18457 ts_tablet_manager.cc:1918] T b770ca6b62834c63872a26b7ae92eb7b P 3339b57741e044b2aed58b42a5b43623: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:30:06.046738 18457 log.cc:1196] T b770ca6b62834c63872a26b7ae92eb7b P 3339b57741e044b2aed58b42a5b43623: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/b770ca6b62834c63872a26b7ae92eb7b
I20260929 16:30:06.047065 18457 ts_tablet_manager.cc:1939] T b770ca6b62834c63872a26b7ae92eb7b P 3339b57741e044b2aed58b42a5b43623: Deleting consensus metadata
I20260929 16:30:06.047883 17952 catalog_manager.cc:4841] TS 3ff55a1ba2bd49de89438c23a80b4ce1 (127.0.98.3:45067): tablet b770ca6b62834c63872a26b7ae92eb7b (table test-table [id=94cf3dbff26f410ab54c974da2cecbbf]) successfully deleted
I20260929 16:30:06.048220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 18157
W20260929 16:30:06.048223 17952 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35103 (error 108)
I20260929 16:30:06.050110 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:06.049707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35103 (local address 127.0.0.1:44630)
0929 16:30:06.049921 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:06.050003 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260929 16:30:06.050350 17952 catalog_manager.cc:4584] TS 7a6ec3a3205b4b1d87c68ef59f30a81a (127.0.98.1:35103): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b770ca6b62834c63872a26b7ae92eb7b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35103: connect: Connection refused (error 111)
I20260929 16:30:06.054314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 18288
I20260929 16:30:06.055938 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:06.054566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40079 (local address 127.0.0.1:49800)
0929 16:30:06.055725 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:06.055799 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1041}
I20260929 16:30:06.060449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 17935
2026-09-29T16:30:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1133 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:06 dist-test-slave-vq9v krb5kdc[18463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:06 dist-test-slave-vq9v krb5kdc[18463](info): set up 2 sockets
Sep 29 16:30:06 dist-test-slave-vq9v krb5kdc[18463](info): commencing operation
krb5kdc: starting...
W20260929 16:30:07.113153 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:07 dist-test-slave-vq9v krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:30:07.245029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38635
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:07.361336 18476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:07.361754 18476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:07.361873 18476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:07.366335 18476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:07.366467 18476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:07.366504 18476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:07.366547 18476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:07.366585 18476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:07.372465 18476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:07.373729 18476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:07.374841 18476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:07.384181 18482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:07.384171 18481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:07.384258 18476 server_base.cc:1034] running on GCE node
W20260929 16:30:07.384171 18484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:07.385062 18476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:07.386328 18476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:07.387547 18476 hybrid_clock.cc:648] HybridClock initialized: now 1790699407387532 us; error 33 us; skew 500 ppm
Sep 29 16:30:07 dist-test-slave-vq9v krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:07.390658 18476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:30:07.391743 18476 webserver.cc:460] Webserver started at http://127.0.98.62:36337/ using document root <none> and password file <none>
I20260929 16:30:07.392364 18476 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:07.392463 18476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:07.392715 18476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:07.394981 18476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "cb1cbe62710e4d5f94e9365d89ed5230"
format_stamp: "Formatted at 2026-09-29 16:30:07 on dist-test-slave-vq9v"
server_key: "4260e16c82a971cef82c214ef6fb3870"
server_key_iv: "0a8a58120ae16eb0a528ecf588605077"
server_key_version: "encryptionkey@0"
I20260929 16:30:07.395602 18476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "cb1cbe62710e4d5f94e9365d89ed5230"
format_stamp: "Formatted at 2026-09-29 16:30:07 on dist-test-slave-vq9v"
server_key: "4260e16c82a971cef82c214ef6fb3870"
server_key_iv: "0a8a58120ae16eb0a528ecf588605077"
server_key_version: "encryptionkey@0"
I20260929 16:30:07.399724 18476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:30:07.402555 18491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:07.404001 18476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:30:07.404130 18476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "cb1cbe62710e4d5f94e9365d89ed5230"
format_stamp: "Formatted at 2026-09-29 16:30:07 on dist-test-slave-vq9v"
server_key: "4260e16c82a971cef82c214ef6fb3870"
server_key_iv: "0a8a58120ae16eb0a528ecf588605077"
server_key_version: "encryptionkey@0"
I20260929 16:30:07.404240 18476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:07.419945 18476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:07.423852 18476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:07.424043 18476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:07.431510 18476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38635
I20260929 16:30:07.431598 18543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38635 every 8 connection(s)
I20260929 16:30:07.432709 18476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:07.435855 18544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:07.442055 18544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230: Bootstrap starting.
I20260929 16:30:07.442446 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 18476
I20260929 16:30:07.442632 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:07.442986 392 external_mini_cluster.cc:1428] Setting key 684acb46a8835be4d2060b64dcd1125a
I20260929 16:30:07.445099 18544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:07.445926 18544 log.cc:824] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:07.447990 18544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230: No bootstrap required, opened a new log
I20260929 16:30:07.451968 18544 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1cbe62710e4d5f94e9365d89ed5230" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38635 } }
I20260929 16:30:07.452270 18544 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:07.452345 18544 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb1cbe62710e4d5f94e9365d89ed5230, State: Initialized, Role: FOLLOWER
Sep 29 16:30:07 dist-test-slave-vq9v krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:30:07.452811 18544 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1cbe62710e4d5f94e9365d89ed5230" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38635 } }
I20260929 16:30:07.452957 18544 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:07.453052 18544 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:07.453187 18544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:07.454650 18544 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1cbe62710e4d5f94e9365d89ed5230" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38635 } }
I20260929 16:30:07.455063 18544 leader_election.cc:302] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb1cbe62710e4d5f94e9365d89ed5230; no voters:
I20260929 16:30:07.455488 18544 leader_election.cc:288] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:07.455574 18549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:07.455754 18549 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230 [term 1 LEADER]: Becoming Leader. State: Replica: cb1cbe62710e4d5f94e9365d89ed5230, State: Running, Role: LEADER
I20260929 16:30:07.456143 18549 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1cbe62710e4d5f94e9365d89ed5230" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38635 } }
I20260929 16:30:07.456892 18544 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:07.457525 18551 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb1cbe62710e4d5f94e9365d89ed5230" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1cbe62710e4d5f94e9365d89ed5230" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38635 } } }
I20260929 16:30:07.457674 18551 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:07.457875 18550 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb1cbe62710e4d5f94e9365d89ed5230. Latest consensus state: current_term: 1 leader_uuid: "cb1cbe62710e4d5f94e9365d89ed5230" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1cbe62710e4d5f94e9365d89ed5230" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38635 } } }
I20260929 16:30:07.457966 18550 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:07.459493 18553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:07.460776 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:07.444537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35550 (local address 127.0.98.62:38635)
0929 16:30:07.445030 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:07.445047 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:07.445113 (+ 66us) server_negotiation.cc:407] Connection header received
0929 16:30:07.445943 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:07.445969 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:07.446403 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:07.446798 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:07.447726 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:07.448799 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:07.449829 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:07.450386 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:07.450580 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:07.453613 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:07.453619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:07.453638 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:07.453683 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:07.456149 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:07.456729 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:07.456734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:07.456739 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:07.456863 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:07.457258 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:07.457262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:07.457263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:07.457946 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:07.458999 (+ 1053us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:07.459170 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 16:30:07.459518 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":342,"thread_start_us":144,"threads_started":1}
I20260929 16:30:07.462775 18553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:07.468434 18553 catalog_manager.cc:1347] Generated new cluster ID: dc30d7d0956f482fa6e249e98a9d9298
I20260929 16:30:07.468525 18553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:07.496779 18553 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:07.498162 18553 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:07.505821 18553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230: Generated new TSK 0
I20260929 16:30:07.506690 18553 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 16:30:07.544577 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38635
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:07.666148 18570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:07.666445 18570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:07.666491 18570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:07.670742 18570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:07.670814 18570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:07.670877 18570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:30:07.676384 18570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:07.677498 18570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:07.678515 18570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:07.688474 18576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:07.688486 18575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:07.688483 18578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:07.688598 18570 server_base.cc:1034] running on GCE node
I20260929 16:30:07.689334 18570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:07.690016 18570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:07.691185 18570 hybrid_clock.cc:648] HybridClock initialized: now 1790699407691171 us; error 26 us; skew 500 ppm
Sep 29 16:30:07 dist-test-slave-vq9v krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:07.694130 18570 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:30:07.695082 18570 webserver.cc:460] Webserver started at http://127.0.98.1:44537/ using document root <none> and password file <none>
I20260929 16:30:07.695602 18570 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:07.695668 18570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:07.695847 18570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:07.698139 18570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "02bb33fadfec4c99a9885ac2ed983a9a"
format_stamp: "Formatted at 2026-09-29 16:30:07 on dist-test-slave-vq9v"
server_key: "0852e9c6e623f3aa728cd55a87d7e280"
server_key_iv: "a9e662f5156d88f32dce398cb0a62b79"
server_key_version: "encryptionkey@0"
I20260929 16:30:07.698918 18570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "02bb33fadfec4c99a9885ac2ed983a9a"
format_stamp: "Formatted at 2026-09-29 16:30:07 on dist-test-slave-vq9v"
server_key: "0852e9c6e623f3aa728cd55a87d7e280"
server_key_iv: "a9e662f5156d88f32dce398cb0a62b79"
server_key_version: "encryptionkey@0"
I20260929 16:30:07.702982 18570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 16:30:07.705667 18585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:07.707168 18570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:07.707314 18570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "02bb33fadfec4c99a9885ac2ed983a9a"
format_stamp: "Formatted at 2026-09-29 16:30:07 on dist-test-slave-vq9v"
server_key: "0852e9c6e623f3aa728cd55a87d7e280"
server_key_iv: "a9e662f5156d88f32dce398cb0a62b79"
server_key_version: "encryptionkey@0"
I20260929 16:30:07.707444 18570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:07.735069 18570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:07.739014 18570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:07.739279 18570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:07.739969 18570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:07.740880 18570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:07.740955 18570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:07.741030 18570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:07.741088 18570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:07.750991 18570 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34901
I20260929 16:30:07.751008 18698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34901 every 8 connection(s)
I20260929 16:30:07.751984 18570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:30:07.762341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 18570
I20260929 16:30:07.762514 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:30:07.762827 392 external_mini_cluster.cc:1428] Setting key 2278c3eccc09d98058a6ff70adfdc8aa
Sep 29 16:30:07 dist-test-slave-vq9v krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:30:07.767867 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:07.753962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47671 (local address 127.0.98.62:38635)
0929 16:30:07.754124 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:07.754128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:07.755343 (+ 1215us) server_negotiation.cc:407] Connection header received
0929 16:30:07.756333 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:07.756337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:07.756397 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:07.756515 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:07.758034 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:07.758734 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:07.760100 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:07.760374 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:07.760503 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:07.763679 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:07.763684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:07.763687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:07.763721 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:07.765532 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:07.766137 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:07.766141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:07.766143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:07.766249 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:07.766631 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:07.766635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:07.766636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:07.766849 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:07.766963 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:07.767546 (+ 583us) server_negotiation.cc:299] Negotiation successful
0929 16:30:07.767687 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:30:07.768782 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:07.754385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38635 (local address 127.0.98.1:47671)
0929 16:30:07.755157 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:07.755204 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:07.756094 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:07.756709 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:07.756718 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:07.757284 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:07.757836 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:07.757851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:07.758877 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:07.758880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:07.759961 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:07.759974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:07.760555 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:07.760559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:07.760755 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:07.761471 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:07.761499 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:07.763442 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:07.765676 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:07.765684 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:07.765712 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:07.766018 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:07.766352 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:07.766355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:07.766357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:07.766525 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:07.766964 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:07.766974 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:07.767289 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 16:30:07.767506 (+ 217us) client_negotiation.cc:240] Negotiation successful
0929 16:30:07.767877 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":133,"threads_started":1}
I20260929 16:30:07.770423 18699 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38635
I20260929 16:30:07.770721 18699 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:07.771288 18699 heartbeater.cc:511] Master 127.0.98.62:38635 requested a full tablet report, sending...
I20260929 16:30:07.773286 18508 ts_manager.cc:194] Registered new tserver with Master: 02bb33fadfec4c99a9885ac2ed983a9a (127.0.98.1:34901)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 16:30:07.774734 18508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 16:30:07.793030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38635
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:07.913075 18704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:07.913427 18704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:07.913473 18704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:07.917666 18704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:07.917747 18704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:07.917810 18704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:30:07.923714 18704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:07.924803 18704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:07.925910 18704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:30:07.936623 18704 server_base.cc:1034] running on GCE node
W20260929 16:30:07.936633 18710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:07.936657 18709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:07.936637 18712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:07.937508 18704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:07.938251 18704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:07.939447 18704 hybrid_clock.cc:648] HybridClock initialized: now 1790699407939399 us; error 63 us; skew 500 ppm
Sep 29 16:30:07 dist-test-slave-vq9v krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:07.942678 18704 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:30:07.943632 18704 webserver.cc:460] Webserver started at http://127.0.98.2:33391/ using document root <none> and password file <none>
I20260929 16:30:07.944214 18704 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:07.944279 18704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:07.944453 18704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:07.946727 18704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "f886984324d14fdebccef18989682e34"
format_stamp: "Formatted at 2026-09-29 16:30:07 on dist-test-slave-vq9v"
server_key: "21c129c90f151942ac8d147450c46f6b"
server_key_iv: "4df220a6078b509112deb129f08235fa"
server_key_version: "encryptionkey@0"
I20260929 16:30:07.947299 18704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f886984324d14fdebccef18989682e34"
format_stamp: "Formatted at 2026-09-29 16:30:07 on dist-test-slave-vq9v"
server_key: "21c129c90f151942ac8d147450c46f6b"
server_key_iv: "4df220a6078b509112deb129f08235fa"
server_key_version: "encryptionkey@0"
I20260929 16:30:07.951429 18704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:30:07.954144 18719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:07.955458 18704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:07.955565 18704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "f886984324d14fdebccef18989682e34"
format_stamp: "Formatted at 2026-09-29 16:30:07 on dist-test-slave-vq9v"
server_key: "21c129c90f151942ac8d147450c46f6b"
server_key_iv: "4df220a6078b509112deb129f08235fa"
server_key_version: "encryptionkey@0"
I20260929 16:30:07.955716 18704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:07.969858 18704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:07.973269 18704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:07.973461 18704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:07.974052 18704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:07.974906 18704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:07.974953 18704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:07.974993 18704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:07.975008 18704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:07.984683 18704 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33155
I20260929 16:30:07.984829 18832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33155 every 8 connection(s)
I20260929 16:30:07.985801 18704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:30:07.990345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 18704
I20260929 16:30:07.990512 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:30:07.990849 392 external_mini_cluster.cc:1428] Setting key 0beb03e3253f336886a73e5e7aee4541
Sep 29 16:30:07 dist-test-slave-vq9v krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 16:30:08.001667 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:07.988055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55039 (local address 127.0.98.62:38635)
0929 16:30:07.988208 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:07.988212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:07.989194 (+ 982us) server_negotiation.cc:407] Connection header received
0929 16:30:07.990224 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:07.990228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:07.990290 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:07.990384 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:07.991935 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:07.992460 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:07.993814 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:07.994046 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:07.994169 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:07.997391 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:07.997395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:07.997397 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:07.997446 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:07.999296 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:07.999899 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:07.999903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:07.999905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:07.999997 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:08.000388 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:08.000392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:08.000395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:08.000617 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:08.000738 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:08.001361 (+ 623us) server_negotiation.cc:299] Negotiation successful
0929 16:30:08.001523 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:30:08.002632 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:07.988472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38635 (local address 127.0.98.2:55039)
0929 16:30:07.989008 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:07.989055 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:07.989984 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:07.990631 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:07.990647 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:07.991134 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:07.991781 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:07.991797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:07.992623 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:07.992632 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:07.993661 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:07.993672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:07.994220 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:07.994224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:07.994431 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:07.995168 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:07.995186 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:07.997035 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:07.999445 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:07.999451 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:07.999472 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:07.999776 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:08.000099 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:08.000103 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:08.000105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:08.000277 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:08.000736 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:08.000746 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:08.001037 (+ 291us) client_negotiation.cc:769] Sending connection context
0929 16:30:08.001282 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 16:30:08.001646 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":148,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 16:30:08.004035 18833 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38635
I20260929 16:30:08.004348 18833 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:08.004989 18833 heartbeater.cc:511] Master 127.0.98.62:38635 requested a full tablet report, sending...
I20260929 16:30:08.006431 18508 ts_manager.cc:194] Registered new tserver with Master: f886984324d14fdebccef18989682e34 (127.0.98.2:33155)
I20260929 16:30:08.007103 18508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 16:30:08.021716 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38635
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:08.138572 18838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:08.138932 18838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:08.139019 18838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:08.143389 18838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:08.143556 18838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:08.143666 18838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:30:08.149660 18838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:08.151185 18838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:08.152271 18838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:08.163152 18843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:08.163156 18844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:08.163156 18846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:08.163365 18838 server_base.cc:1034] running on GCE node
I20260929 16:30:08.163997 18838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:08.164759 18838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:08.165983 18838 hybrid_clock.cc:648] HybridClock initialized: now 1790699408165940 us; error 57 us; skew 500 ppm
Sep 29 16:30:08 dist-test-slave-vq9v krb5kdc[18463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:08.169279 18838 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:30:08.170454 18838 webserver.cc:460] Webserver started at http://127.0.98.3:39329/ using document root <none> and password file <none>
I20260929 16:30:08.171118 18838 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:08.171221 18838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:08.171463 18838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:08.173799 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "05ddeeb6d1dd4dc2aaf1577dede5f95c"
format_stamp: "Formatted at 2026-09-29 16:30:08 on dist-test-slave-vq9v"
server_key: "abf79483553b0a3e550dcb0c4ca8c641"
server_key_iv: "4edacdc391188627e48ced30065e5cb4"
server_key_version: "encryptionkey@0"
I20260929 16:30:08.174476 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "05ddeeb6d1dd4dc2aaf1577dede5f95c"
format_stamp: "Formatted at 2026-09-29 16:30:08 on dist-test-slave-vq9v"
server_key: "abf79483553b0a3e550dcb0c4ca8c641"
server_key_iv: "4edacdc391188627e48ced30065e5cb4"
server_key_version: "encryptionkey@0"
I20260929 16:30:08.178817 18838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:30:08.181701 18853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:08.183153 18838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:30:08.183324 18838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "05ddeeb6d1dd4dc2aaf1577dede5f95c"
format_stamp: "Formatted at 2026-09-29 16:30:08 on dist-test-slave-vq9v"
server_key: "abf79483553b0a3e550dcb0c4ca8c641"
server_key_iv: "4edacdc391188627e48ced30065e5cb4"
server_key_version: "encryptionkey@0"
I20260929 16:30:08.183468 18838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:08.198495 18838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:08.202082 18838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:08.202302 18838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:08.202924 18838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:08.204180 18838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:08.204262 18838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:08.204331 18838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:08.204365 18838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:08.215946 18838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37877
I20260929 16:30:08.216050 18966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37877 every 8 connection(s)
I20260929 16:30:08.216910 18838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:30:08.218868 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 18838
I20260929 16:30:08.218991 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:30:08.219313 392 external_mini_cluster.cc:1428] Setting key 81ddbea97f1120147f27e1266682ec6b
Sep 29 16:30:08 dist-test-slave-vq9v krb5kdc[18463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:30:08.231040 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.218895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52133 (local address 127.0.98.62:38635)
0929 16:30:08.219030 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:08.219037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:08.220171 (+ 1134us) server_negotiation.cc:407] Connection header received
0929 16:30:08.221423 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:08.221426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:08.221500 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:08.221584 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:08.222774 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.223289 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.224384 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.224592 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.224710 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.227330 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:08.227334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:08.227336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:08.227361 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:08.228767 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:08.229404 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:08.229407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:08.229408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:08.229491 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:08.229836 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:08.229838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:08.229839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:08.230027 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:08.230104 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:08.230758 (+ 654us) server_negotiation.cc:299] Negotiation successful
0929 16:30:08.230886 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:30:08.231999 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.219438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38635 (local address 127.0.98.3:52133)
0929 16:30:08.219987 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:08.220054 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:08.221143 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:08.221773 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:08.221781 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:08.222153 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:08.222642 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:08.222656 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.223395 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.223398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:08.224279 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:08.224288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.224715 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.224718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:08.224851 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.225511 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:08.225526 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:08.227131 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:08.228897 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:08.228903 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:08.228917 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:08.229295 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:08.229581 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:08.229583 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:08.229585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:08.229742 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:08.230160 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:08.230167 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:08.230470 (+ 303us) client_negotiation.cc:769] Sending connection context
0929 16:30:08.230659 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 16:30:08.230962 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":144,"threads_started":1}
I20260929 16:30:08.233446 18967 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38635
I20260929 16:30:08.233757 18967 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:08.234378 18967 heartbeater.cc:511] Master 127.0.98.62:38635 requested a full tablet report, sending...
I20260929 16:30:08.235462 18508 ts_manager.cc:194] Registered new tserver with Master: 05ddeeb6d1dd4dc2aaf1577dede5f95c (127.0.98.3:37877)
I20260929 16:30:08.236176 18508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52133
I20260929 16:30:08.246266 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:30:08.256568 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.248385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35564 (local address 127.0.98.62:38635)
0929 16:30:08.248524 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:08.248528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:08.248740 (+ 212us) server_negotiation.cc:407] Connection header received
0929 16:30:08.248818 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:08.248820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:08.248862 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:08.248936 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:08.249789 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.250313 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.251171 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.251371 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.251448 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.253176 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:08.253182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:08.253184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:08.253239 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:08.254679 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:08.255210 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:08.255213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:08.255215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:08.255301 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:08.255640 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:08.255643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:08.255651 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:08.255867 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:08.256029 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:08.256306 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 16:30:08.256405 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:30:08.260157 18508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:30:08.263682 18508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:30:08.278982 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.272855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37877 (local address 127.0.0.1:47756)
0929 16:30:08.273175 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:08.273277 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:08.273613 (+ 336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:08.274469 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:08.274477 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:08.274503 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:08.274664 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:08.274671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.275826 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.275830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:08.277242 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:08.277253 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.278385 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.278389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:08.278505 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.278552 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 16:30:08.278664 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 16:30:08.278809 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":151,"thread_start_us":89,"threads_started":1}
I20260929 16:30:08.279178 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.274421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54814 (local address 127.0.98.2:33155)
0929 16:30:08.274784 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:08.274795 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:08.274815 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:30:08.274880 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:08.274884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:08.275035 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:08.275453 (+ 418us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:08.275608 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.276657 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.277896 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.278559 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.278672 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.278844 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:08.278900 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 16:30:08.278996 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":146,"threads_started":1}
I20260929 16:30:08.279536 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.273078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.0.98.3:37877)
0929 16:30:08.273517 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:08.273531 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:08.273557 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:30:08.273841 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:08.273847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:08.274081 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:08.274244 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:08.274776 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.275692 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.277372 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.278247 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.278376 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.278539 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:08.279281 (+ 742us) server_negotiation.cc:299] Negotiation successful
0929 16:30:08.279415 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":202,"threads_started":1}
I20260929 16:30:08.279140 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.273970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33155 (local address 127.0.0.1:54814)
0929 16:30:08.274411 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:08.274424 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:08.274487 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:08.275193 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:08.275199 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:08.275217 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:08.275311 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:08.275317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.276781 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.276785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:08.277768 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:08.277776 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.278684 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.278687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:08.278776 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.278800 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:30:08.278858 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 16:30:08.278928 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":20,"thread_start_us":84,"threads_started":1}
I20260929 16:30:08.279989 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.273678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34901 (local address 127.0.0.1:48904)
0929 16:30:08.274369 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:08.274383 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:08.274485 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:08.275024 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:08.275027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:08.275047 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:08.275137 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:08.275142 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.277028 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.277033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:08.278280 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:08.278293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.279546 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.279548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:08.279695 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.279718 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:30:08.279786 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 16:30:08.279853 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":47,"thread_start_us":80,"threads_started":1}
I20260929 16:30:08.280388 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.273860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48904 (local address 127.0.98.1:34901)
0929 16:30:08.274223 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:08.274229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:08.274589 (+ 360us) server_negotiation.cc:407] Connection header received
0929 16:30:08.274689 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:08.274694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:08.274884 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:08.275029 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:08.275233 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.276625 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.278417 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.279265 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.279363 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.279493 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:08.280143 (+ 650us) server_negotiation.cc:299] Negotiation successful
0929 16:30:08.280249 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":84,"threads_started":1}
I20260929 16:30:08.282778 18767 tablet_service.cc:1467] Processing CreateTablet for tablet 59727a30d6a84c8c948c1f745ddbb265 (DEFAULT_TABLE table=test-table [id=c712696868f549f1afdd1fc211a8cb18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:08.282898 18633 tablet_service.cc:1467] Processing CreateTablet for tablet 59727a30d6a84c8c948c1f745ddbb265 (DEFAULT_TABLE table=test-table [id=c712696868f549f1afdd1fc211a8cb18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:08.282902 18899 tablet_service.cc:1467] Processing CreateTablet for tablet 59727a30d6a84c8c948c1f745ddbb265 (DEFAULT_TABLE table=test-table [id=c712696868f549f1afdd1fc211a8cb18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:08.283682 18767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59727a30d6a84c8c948c1f745ddbb265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:08.283891 18633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59727a30d6a84c8c948c1f745ddbb265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:08.283891 18899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59727a30d6a84c8c948c1f745ddbb265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:08.290249 18984 tablet_bootstrap.cc:492] T 59727a30d6a84c8c948c1f745ddbb265 P f886984324d14fdebccef18989682e34: Bootstrap starting.
I20260929 16:30:08.291587 18986 tablet_bootstrap.cc:492] T 59727a30d6a84c8c948c1f745ddbb265 P 05ddeeb6d1dd4dc2aaf1577dede5f95c: Bootstrap starting.
I20260929 16:30:08.292450 18984 tablet_bootstrap.cc:654] T 59727a30d6a84c8c948c1f745ddbb265 P f886984324d14fdebccef18989682e34: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:08.293390 18986 tablet_bootstrap.cc:654] T 59727a30d6a84c8c948c1f745ddbb265 P 05ddeeb6d1dd4dc2aaf1577dede5f95c: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:08.293576 18985 tablet_bootstrap.cc:492] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a: Bootstrap starting.
I20260929 16:30:08.293597 18984 log.cc:824] T 59727a30d6a84c8c948c1f745ddbb265 P f886984324d14fdebccef18989682e34: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:08.294658 18986 log.cc:824] T 59727a30d6a84c8c948c1f745ddbb265 P 05ddeeb6d1dd4dc2aaf1577dede5f95c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:08.295663 18984 tablet_bootstrap.cc:492] T 59727a30d6a84c8c948c1f745ddbb265 P f886984324d14fdebccef18989682e34: No bootstrap required, opened a new log
I20260929 16:30:08.295866 18985 tablet_bootstrap.cc:654] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:08.295925 18984 ts_tablet_manager.cc:1397] T 59727a30d6a84c8c948c1f745ddbb265 P f886984324d14fdebccef18989682e34: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:30:08.296566 18985 log.cc:824] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:08.298537 18985 tablet_bootstrap.cc:492] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a: No bootstrap required, opened a new log
I20260929 16:30:08.298759 18985 ts_tablet_manager.cc:1397] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:30:08.298940 18986 tablet_bootstrap.cc:492] T 59727a30d6a84c8c948c1f745ddbb265 P 05ddeeb6d1dd4dc2aaf1577dede5f95c: No bootstrap required, opened a new log
I20260929 16:30:08.299145 18986 ts_tablet_manager.cc:1397] T 59727a30d6a84c8c948c1f745ddbb265 P 05ddeeb6d1dd4dc2aaf1577dede5f95c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 16:30:08.299806 18984 raft_consensus.cc:348] T 59727a30d6a84c8c948c1f745ddbb265 P f886984324d14fdebccef18989682e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ddeeb6d1dd4dc2aaf1577dede5f95c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37877 } } peers { permanent_uuid: "02bb33fadfec4c99a9885ac2ed983a9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34901 } } peers { permanent_uuid: "f886984324d14fdebccef18989682e34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33155 } }
I20260929 16:30:08.300117 18984 raft_consensus.cc:374] T 59727a30d6a84c8c948c1f745ddbb265 P f886984324d14fdebccef18989682e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:08.300181 18984 raft_consensus.cc:729] T 59727a30d6a84c8c948c1f745ddbb265 P f886984324d14fdebccef18989682e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f886984324d14fdebccef18989682e34, State: Initialized, Role: FOLLOWER
I20260929 16:30:08.300659 18984 consensus_queue.cc:260] T 59727a30d6a84c8c948c1f745ddbb265 P f886984324d14fdebccef18989682e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ddeeb6d1dd4dc2aaf1577dede5f95c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37877 } } peers { permanent_uuid: "02bb33fadfec4c99a9885ac2ed983a9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34901 } } peers { permanent_uuid: "f886984324d14fdebccef18989682e34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33155 } }
I20260929 16:30:08.301410 18833 heartbeater.cc:503] Master 127.0.98.62:38635 was elected leader, sending a full tablet report...
I20260929 16:30:08.301774 18984 ts_tablet_manager.cc:1428] T 59727a30d6a84c8c948c1f745ddbb265 P f886984324d14fdebccef18989682e34: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:30:08.303112 18985 raft_consensus.cc:348] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ddeeb6d1dd4dc2aaf1577dede5f95c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37877 } } peers { permanent_uuid: "02bb33fadfec4c99a9885ac2ed983a9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34901 } } peers { permanent_uuid: "f886984324d14fdebccef18989682e34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33155 } }
I20260929 16:30:08.303365 18985 raft_consensus.cc:374] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:08.303220 18986 raft_consensus.cc:348] T 59727a30d6a84c8c948c1f745ddbb265 P 05ddeeb6d1dd4dc2aaf1577dede5f95c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ddeeb6d1dd4dc2aaf1577dede5f95c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37877 } } peers { permanent_uuid: "02bb33fadfec4c99a9885ac2ed983a9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34901 } } peers { permanent_uuid: "f886984324d14fdebccef18989682e34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33155 } }
I20260929 16:30:08.303421 18985 raft_consensus.cc:729] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02bb33fadfec4c99a9885ac2ed983a9a, State: Initialized, Role: FOLLOWER
I20260929 16:30:08.303480 18986 raft_consensus.cc:374] T 59727a30d6a84c8c948c1f745ddbb265 P 05ddeeb6d1dd4dc2aaf1577dede5f95c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:08.303539 18986 raft_consensus.cc:729] T 59727a30d6a84c8c948c1f745ddbb265 P 05ddeeb6d1dd4dc2aaf1577dede5f95c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ddeeb6d1dd4dc2aaf1577dede5f95c, State: Initialized, Role: FOLLOWER
I20260929 16:30:08.303881 18985 consensus_queue.cc:260] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ddeeb6d1dd4dc2aaf1577dede5f95c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37877 } } peers { permanent_uuid: "02bb33fadfec4c99a9885ac2ed983a9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34901 } } peers { permanent_uuid: "f886984324d14fdebccef18989682e34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33155 } }
I20260929 16:30:08.303962 18986 consensus_queue.cc:260] T 59727a30d6a84c8c948c1f745ddbb265 P 05ddeeb6d1dd4dc2aaf1577dede5f95c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ddeeb6d1dd4dc2aaf1577dede5f95c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37877 } } peers { permanent_uuid: "02bb33fadfec4c99a9885ac2ed983a9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34901 } } peers { permanent_uuid: "f886984324d14fdebccef18989682e34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33155 } }
I20260929 16:30:08.304533 18699 heartbeater.cc:503] Master 127.0.98.62:38635 was elected leader, sending a full tablet report...
I20260929 16:30:08.304896 18985 ts_tablet_manager.cc:1428] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:30:08.304927 18986 ts_tablet_manager.cc:1428] T 59727a30d6a84c8c948c1f745ddbb265 P 05ddeeb6d1dd4dc2aaf1577dede5f95c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:30:08.305272 18967 heartbeater.cc:503] Master 127.0.98.62:38635 was elected leader, sending a full tablet report...
I20260929 16:30:08.404098 18991 raft_consensus.cc:482] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:30:08.404330 18991 raft_consensus.cc:504] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ddeeb6d1dd4dc2aaf1577dede5f95c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37877 } } peers { permanent_uuid: "02bb33fadfec4c99a9885ac2ed983a9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34901 } } peers { permanent_uuid: "f886984324d14fdebccef18989682e34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33155 } }
I20260929 16:30:08.405530 18991 leader_election.cc:288] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05ddeeb6d1dd4dc2aaf1577dede5f95c (127.0.98.3:37877), f886984324d14fdebccef18989682e34 (127.0.98.2:33155)
I20260929 16:30:08.410249 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.405988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37177 (local address 127.0.98.3:37877)
0929 16:30:08.406142 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:08.406146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:08.406422 (+ 276us) server_negotiation.cc:407] Connection header received
0929 16:30:08.406585 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:08.406588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:08.406652 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:08.406803 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:08.407202 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.407921 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.409159 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.409651 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.409742 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.409794 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:08.410026 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 16:30:08.410116 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:30:08.410279 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.405937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37877 (local address 127.0.98.1:37177)
0929 16:30:08.406307 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:08.406333 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:08.406455 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:08.406820 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:08.406823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:08.406840 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:08.407062 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:08.407070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.408035 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.408039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:08.409017 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:08.409028 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.409786 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.409792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:08.409893 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.409916 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:30:08.409990 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 16:30:08.410062 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"spinlock_wait_cycles":37120,"thread_start_us":146,"threads_started":1}
I20260929 16:30:08.410316 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.406538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35401 (local address 127.0.98.2:33155)
0929 16:30:08.406680 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:08.406683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:08.406911 (+ 228us) server_negotiation.cc:407] Connection header received
0929 16:30:08.407068 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:08.407071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:08.407129 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:08.407207 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:08.407537 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.408071 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.409294 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.409755 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.409858 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.409922 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:08.410128 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 16:30:08.410203 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 16:30:08.410732 18994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.406388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33155 (local address 127.0.98.1:35401)
0929 16:30:08.406803 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:08.406817 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:08.406958 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:08.407235 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:08.407238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:08.407254 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:08.407386 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:08.407393 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.408197 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.408200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:08.409137 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:08.409151 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.409885 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.409888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:08.409996 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.410020 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:30:08.410097 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 16:30:08.410160 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"spinlock_wait_cycles":1792,"thread_start_us":93,"threads_started":1}
I20260929 16:30:08.411226 18920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59727a30d6a84c8c948c1f745ddbb265" candidate_uuid: "02bb33fadfec4c99a9885ac2ed983a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ddeeb6d1dd4dc2aaf1577dede5f95c" is_pre_election: true
I20260929 16:30:08.411453 18787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59727a30d6a84c8c948c1f745ddbb265" candidate_uuid: "02bb33fadfec4c99a9885ac2ed983a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f886984324d14fdebccef18989682e34" is_pre_election: true
I20260929 16:30:08.411701 18920 raft_consensus.cc:2454] T 59727a30d6a84c8c948c1f745ddbb265 P 05ddeeb6d1dd4dc2aaf1577dede5f95c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02bb33fadfec4c99a9885ac2ed983a9a in term 0.
I20260929 16:30:08.411712 18787 raft_consensus.cc:2454] T 59727a30d6a84c8c948c1f745ddbb265 P f886984324d14fdebccef18989682e34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02bb33fadfec4c99a9885ac2ed983a9a in term 0.
I20260929 16:30:08.412264 18588 leader_election.cc:302] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02bb33fadfec4c99a9885ac2ed983a9a, f886984324d14fdebccef18989682e34; no voters:
I20260929 16:30:08.412547 18991 raft_consensus.cc:2781] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:30:08.412675 18991 raft_consensus.cc:482] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:30:08.412729 18991 raft_consensus.cc:3037] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:08.413928 18991 raft_consensus.cc:504] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ddeeb6d1dd4dc2aaf1577dede5f95c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37877 } } peers { permanent_uuid: "02bb33fadfec4c99a9885ac2ed983a9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34901 } } peers { permanent_uuid: "f886984324d14fdebccef18989682e34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33155 } }
I20260929 16:30:08.414331 18991 leader_election.cc:288] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a [CANDIDATE]: Term 1 election: Requested vote from peers 05ddeeb6d1dd4dc2aaf1577dede5f95c (127.0.98.3:37877), f886984324d14fdebccef18989682e34 (127.0.98.2:33155)
I20260929 16:30:08.414911 18920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59727a30d6a84c8c948c1f745ddbb265" candidate_uuid: "02bb33fadfec4c99a9885ac2ed983a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ddeeb6d1dd4dc2aaf1577dede5f95c"
I20260929 16:30:08.414911 18787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59727a30d6a84c8c948c1f745ddbb265" candidate_uuid: "02bb33fadfec4c99a9885ac2ed983a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f886984324d14fdebccef18989682e34"
I20260929 16:30:08.415060 18787 raft_consensus.cc:3037] T 59727a30d6a84c8c948c1f745ddbb265 P f886984324d14fdebccef18989682e34 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:08.415060 18920 raft_consensus.cc:3037] T 59727a30d6a84c8c948c1f745ddbb265 P 05ddeeb6d1dd4dc2aaf1577dede5f95c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:08.416469 18920 raft_consensus.cc:2454] T 59727a30d6a84c8c948c1f745ddbb265 P 05ddeeb6d1dd4dc2aaf1577dede5f95c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02bb33fadfec4c99a9885ac2ed983a9a in term 1.
I20260929 16:30:08.416469 18787 raft_consensus.cc:2454] T 59727a30d6a84c8c948c1f745ddbb265 P f886984324d14fdebccef18989682e34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02bb33fadfec4c99a9885ac2ed983a9a in term 1.
I20260929 16:30:08.416860 18588 leader_election.cc:302] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02bb33fadfec4c99a9885ac2ed983a9a, f886984324d14fdebccef18989682e34; no voters:
I20260929 16:30:08.417059 18991 raft_consensus.cc:2781] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:08.417371 18991 raft_consensus.cc:686] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a [term 1 LEADER]: Becoming Leader. State: Replica: 02bb33fadfec4c99a9885ac2ed983a9a, State: Running, Role: LEADER
I20260929 16:30:08.417749 18991 consensus_queue.cc:237] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ddeeb6d1dd4dc2aaf1577dede5f95c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37877 } } peers { permanent_uuid: "02bb33fadfec4c99a9885ac2ed983a9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34901 } } peers { permanent_uuid: "f886984324d14fdebccef18989682e34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33155 } }
I20260929 16:30:08.421402 18506 catalog_manager.cc:5495] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 02bb33fadfec4c99a9885ac2ed983a9a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "02bb33fadfec4c99a9885ac2ed983a9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05ddeeb6d1dd4dc2aaf1577dede5f95c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02bb33fadfec4c99a9885ac2ed983a9a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f886984324d14fdebccef18989682e34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33155 } health_report { overall_health: UNKNOWN } } }
I20260929 16:30:08.434581 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.430293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48908 (local address 127.0.98.1:34901)
0929 16:30:08.430422 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:08.430426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:08.430439 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:30:08.430509 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:08.430511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:08.430562 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:08.430701 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:30:08.430958 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.431707 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.432712 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.433057 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.433179 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.433537 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:30:08.434043 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:30:08.434212 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:08.434374 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 16:30:08.434423 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:30:08.441627 18787 raft_consensus.cc:1261] T 59727a30d6a84c8c948c1f745ddbb265 P f886984324d14fdebccef18989682e34 [term 1 FOLLOWER]: Refusing update from remote peer 02bb33fadfec4c99a9885ac2ed983a9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:08.441659 18920 raft_consensus.cc:1261] T 59727a30d6a84c8c948c1f745ddbb265 P 05ddeeb6d1dd4dc2aaf1577dede5f95c [term 1 FOLLOWER]: Refusing update from remote peer 02bb33fadfec4c99a9885ac2ed983a9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:08.442421 18991 consensus_queue.cc:1035] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ddeeb6d1dd4dc2aaf1577dede5f95c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:08.442732 18995 consensus_queue.cc:1035] T 59727a30d6a84c8c948c1f745ddbb265 P 02bb33fadfec4c99a9885ac2ed983a9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f886984324d14fdebccef18989682e34" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:08.453783 18999 mvcc.cc:205] Tried to move back new op lower bound from 7334704776968728576 to 7334704776881946624. Current Snapshot: MvccSnapshot[applied={T|T < 7334704776968728576}]
I20260929 16:30:08.460263 18507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:30:08.460506 18507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:30:08.463274 18507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb1cbe62710e4d5f94e9365d89ed5230: Sending DeleteTablet for 3 replicas of tablet 59727a30d6a84c8c948c1f745ddbb265
I20260929 16:30:08.464092 18899 tablet_service.cc:1514] Processing DeleteTablet for tablet 59727a30d6a84c8c948c1f745ddbb265 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47756
I20260929 16:30:08.464190 18767 tablet_service.cc:1514] Processing DeleteTablet for tablet 59727a30d6a84c8c948c1f745ddbb265 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54814
I20260929 16:30:08.464628 19008 tablet_replica.cc:330] T 59727a30d6a84c8c948c1f745ddbb265 P f886984324d14fdebccef18989682e34: stopping tablet replica
I20260929 16:30:08.464788 18633 tablet_service.cc:1514] Processing DeleteTablet for tablet 59727a30d6a84c8c948c1f745ddbb265 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48904
I20260929 16:30:08.465020 19008 raft_consensus.cc:2229] T 59727a30d6a84c8c948c1f745ddbb265 P f886984324d14fdebccef18989682e34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:30:08.465157 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 18570
I20260929 16:30:08.465377 19008 raft_consensus.cc:2258] T 59727a30d6a84c8c948c1f745ddbb265 P f886984324d14fdebccef18989682e34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:08.465806 19009 tablet_replica.cc:330] T 59727a30d6a84c8c948c1f745ddbb265 P 05ddeeb6d1dd4dc2aaf1577dede5f95c: stopping tablet replica
I20260929 16:30:08.466089 19009 raft_consensus.cc:2229] T 59727a30d6a84c8c948c1f745ddbb265 P 05ddeeb6d1dd4dc2aaf1577dede5f95c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:30:08.466382 19009 raft_consensus.cc:2258] T 59727a30d6a84c8c948c1f745ddbb265 P 05ddeeb6d1dd4dc2aaf1577dede5f95c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:08.468165 19009 ts_tablet_manager.cc:1905] T 59727a30d6a84c8c948c1f745ddbb265 P 05ddeeb6d1dd4dc2aaf1577dede5f95c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:30:08.468338 19008 ts_tablet_manager.cc:1905] T 59727a30d6a84c8c948c1f745ddbb265 P f886984324d14fdebccef18989682e34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:30:08.470795 19008 ts_tablet_manager.cc:1918] T 59727a30d6a84c8c948c1f745ddbb265 P f886984324d14fdebccef18989682e34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:30:08.470927 19008 log.cc:1196] T 59727a30d6a84c8c948c1f745ddbb265 P f886984324d14fdebccef18989682e34: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/59727a30d6a84c8c948c1f745ddbb265
I20260929 16:30:08.471840 19009 ts_tablet_manager.cc:1918] T 59727a30d6a84c8c948c1f745ddbb265 P 05ddeeb6d1dd4dc2aaf1577dede5f95c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:30:08.471997 19008 ts_tablet_manager.cc:1939] T 59727a30d6a84c8c948c1f745ddbb265 P f886984324d14fdebccef18989682e34: Deleting consensus metadata
I20260929 16:30:08.472131 19009 log.cc:1196] T 59727a30d6a84c8c948c1f745ddbb265 P 05ddeeb6d1dd4dc2aaf1577dede5f95c: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/59727a30d6a84c8c948c1f745ddbb265
I20260929 16:30:08.472482 19009 ts_tablet_manager.cc:1939] T 59727a30d6a84c8c948c1f745ddbb265 P 05ddeeb6d1dd4dc2aaf1577dede5f95c: Deleting consensus metadata
I20260929 16:30:08.473847 18494 catalog_manager.cc:4841] TS f886984324d14fdebccef18989682e34 (127.0.98.2:33155): tablet 59727a30d6a84c8c948c1f745ddbb265 (table test-table [id=c712696868f549f1afdd1fc211a8cb18]) successfully deleted
W20260929 16:30:08.474198 18494 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34901 (error 108)
I20260929 16:30:08.474319 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 18704
I20260929 16:30:08.474749 18494 catalog_manager.cc:4841] TS 05ddeeb6d1dd4dc2aaf1577dede5f95c (127.0.98.3:37877): tablet 59727a30d6a84c8c948c1f745ddbb265 (table test-table [id=c712696868f549f1afdd1fc211a8cb18]) successfully deleted
I20260929 16:30:08.479450 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.476936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34901 (local address 127.0.0.1:48922)
0929 16:30:08.479282 (+ 2346us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:08.479362 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2056}
W20260929 16:30:08.481377 18494 catalog_manager.cc:4584] TS 02bb33fadfec4c99a9885ac2ed983a9a (127.0.98.1:34901): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 59727a30d6a84c8c948c1f745ddbb265: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34901: connect: Connection refused (error 111)
I20260929 16:30:08.483317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 18838
I20260929 16:30:08.491775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 18476
2026-09-29T16:30:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2437 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T16:30:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:08Z Disabled control of system clock
I20260929 16:30:08.531137 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46671
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:30:08.642467 19015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:08.642779 19015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:08.642818 19015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:08.647032 19015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:08.647106 19015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:08.647122 19015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:08.647140 19015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:08.647161 19015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:08.652271 19015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46671
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:08.653512 19015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:08.654991 19015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:08.664341 19020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:08.664458 19015 server_base.cc:1034] running on GCE node
W20260929 16:30:08.664708 19023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:08.664777 19021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:08.665501 19015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:08.666682 19015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:08.667876 19015 hybrid_clock.cc:648] HybridClock initialized: now 1790699408667847 us; error 46 us; skew 500 ppm
I20260929 16:30:08.670090 19015 webserver.cc:460] Webserver started at http://127.0.98.62:46801/ using document root <none> and password file <none>
I20260929 16:30:08.670771 19015 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:08.670842 19015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:08.671097 19015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:08.673364 19015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "2314f19bfe0049b99da8f2ee103b1c13"
format_stamp: "Formatted at 2026-09-29 16:30:08 on dist-test-slave-vq9v"
server_key: "8750707bbc8dcafde8b4fd1024a2569e"
server_key_iv: "f907d83905ac1539a5a76925f12dcd12"
server_key_version: "encryptionkey@0"
I20260929 16:30:08.673987 19015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "2314f19bfe0049b99da8f2ee103b1c13"
format_stamp: "Formatted at 2026-09-29 16:30:08 on dist-test-slave-vq9v"
server_key: "8750707bbc8dcafde8b4fd1024a2569e"
server_key_iv: "f907d83905ac1539a5a76925f12dcd12"
server_key_version: "encryptionkey@0"
I20260929 16:30:08.678027 19015 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 16:30:08.680732 19029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:08.682178 19015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:30:08.682337 19015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "2314f19bfe0049b99da8f2ee103b1c13"
format_stamp: "Formatted at 2026-09-29 16:30:08 on dist-test-slave-vq9v"
server_key: "8750707bbc8dcafde8b4fd1024a2569e"
server_key_iv: "f907d83905ac1539a5a76925f12dcd12"
server_key_version: "encryptionkey@0"
I20260929 16:30:08.682469 19015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:08.700210 19015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:08.701009 19015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:08.701308 19015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:08.709122 19015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46671
I20260929 16:30:08.709142 19081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46671 every 8 connection(s)
I20260929 16:30:08.710258 19015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:08.713537 19082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:08.718003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 19015
I20260929 16:30:08.718117 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:08.718441 392 external_mini_cluster.cc:1428] Setting key ad7a5a5196a7e0d7c29ed73a0e887cb4
I20260929 16:30:08.720072 19082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13: Bootstrap starting.
I20260929 16:30:08.722464 19082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:08.723290 19082 log.cc:824] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:08.725410 19082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13: No bootstrap required, opened a new log
I20260929 16:30:08.729377 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.719940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57522 (local address 127.0.98.62:46671)
0929 16:30:08.720761 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:08.720777 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:08.720816 (+ 39us) server_negotiation.cc:407] Connection header received
0929 16:30:08.721577 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:08.721599 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:08.722032 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:08.722402 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:08.722880 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.723931 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.724901 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.725636 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.725827 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.726662 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:08.726669 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:08.726687 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:08.727221 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:08.727295 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:08.727324 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:08.727430 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:08.727582 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 16:30:08.727866 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":179,"threads_started":1}
I20260929 16:30:08.729581 19082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2314f19bfe0049b99da8f2ee103b1c13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46671 } }
I20260929 16:30:08.729884 19082 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:08.729940 19082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2314f19bfe0049b99da8f2ee103b1c13, State: Initialized, Role: FOLLOWER
I20260929 16:30:08.730414 19082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2314f19bfe0049b99da8f2ee103b1c13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46671 } }
I20260929 16:30:08.730608 19082 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:08.730679 19082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:08.730786 19082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:08.731880 19082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2314f19bfe0049b99da8f2ee103b1c13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46671 } }
I20260929 16:30:08.732249 19082 leader_election.cc:302] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2314f19bfe0049b99da8f2ee103b1c13; no voters:
I20260929 16:30:08.732604 19082 leader_election.cc:288] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:08.732709 19087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:08.732976 19087 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13 [term 1 LEADER]: Becoming Leader. State: Replica: 2314f19bfe0049b99da8f2ee103b1c13, State: Running, Role: LEADER
I20260929 16:30:08.733366 19087 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2314f19bfe0049b99da8f2ee103b1c13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46671 } }
I20260929 16:30:08.734009 19082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:08.735237 19089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2314f19bfe0049b99da8f2ee103b1c13. Latest consensus state: current_term: 1 leader_uuid: "2314f19bfe0049b99da8f2ee103b1c13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2314f19bfe0049b99da8f2ee103b1c13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46671 } } }
I20260929 16:30:08.735253 19088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2314f19bfe0049b99da8f2ee103b1c13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2314f19bfe0049b99da8f2ee103b1c13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46671 } } }
I20260929 16:30:08.735388 19088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:08.735388 19089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:08.735834 19096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:08.738600 19096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:08.744949 19096 catalog_manager.cc:1347] Generated new cluster ID: fd05b6308686497d8566906348c8a036
I20260929 16:30:08.745055 19096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:08.758258 19096 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:08.759317 19096 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:08.768234 19096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13: Generated new TSK 0
I20260929 16:30:08.768996 19096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:08.775017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46671
--builtin_ntp_servers=127.0.98.20:43649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:30:08.892652 19106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:08.892947 19106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:08.893015 19106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:08.897397 19106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:08.897495 19106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:08.897641 19106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:30:08.902740 19106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:08.903867 19106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:08.905272 19106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:08.915711 19112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:08.915711 19111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:08.915711 19114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:08.915764 19106 server_base.cc:1034] running on GCE node
I20260929 16:30:08.916321 19106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:08.917016 19106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:08.918200 19106 hybrid_clock.cc:648] HybridClock initialized: now 1790699408918169 us; error 43 us; skew 500 ppm
I20260929 16:30:08.927613 19106 webserver.cc:460] Webserver started at http://127.0.98.1:33395/ using document root <none> and password file <none>
I20260929 16:30:08.928277 19106 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:08.928347 19106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:08.928519 19106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:08.930826 19106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "2c49d0a3d222431fa732888acc20c6c5"
format_stamp: "Formatted at 2026-09-29 16:30:08 on dist-test-slave-vq9v"
server_key: "e161b9da94dd534bc1de410912dd7bca"
server_key_iv: "1da5078d318533c0093b293ba0d4369b"
server_key_version: "encryptionkey@0"
I20260929 16:30:08.931394 19106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2c49d0a3d222431fa732888acc20c6c5"
format_stamp: "Formatted at 2026-09-29 16:30:08 on dist-test-slave-vq9v"
server_key: "e161b9da94dd534bc1de410912dd7bca"
server_key_iv: "1da5078d318533c0093b293ba0d4369b"
server_key_version: "encryptionkey@0"
I20260929 16:30:08.935848 19106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:30:08.938939 19120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:08.940361 19106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:30:08.940480 19106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "2c49d0a3d222431fa732888acc20c6c5"
format_stamp: "Formatted at 2026-09-29 16:30:08 on dist-test-slave-vq9v"
server_key: "e161b9da94dd534bc1de410912dd7bca"
server_key_iv: "1da5078d318533c0093b293ba0d4369b"
server_key_version: "encryptionkey@0"
I20260929 16:30:08.940666 19106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:08.970137 19106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:08.970927 19106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:08.971158 19106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:08.971935 19106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:08.972966 19106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:08.973014 19106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:08.973057 19106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:08.973073 19106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:08.982822 19106 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45907
I20260929 16:30:08.982854 19233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45907 every 8 connection(s)
I20260929 16:30:08.984017 19106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:30:08.992806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 19106
I20260929 16:30:08.992970 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:30:08.993366 392 external_mini_cluster.cc:1428] Setting key cb4b93f0bef77961ebf46b2338f751e0
I20260929 16:30:08.994032 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.986144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51909 (local address 127.0.98.62:46671)
0929 16:30:08.986282 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:08.986286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:08.987517 (+ 1231us) server_negotiation.cc:407] Connection header received
0929 16:30:08.988966 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:08.988970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:08.989030 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:08.989132 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:08.990063 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.990600 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.991774 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.991998 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.992111 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.993434 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:08.993438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:08.993442 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:08.993461 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:08.993484 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:08.993494 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:08.993565 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:08.993788 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 16:30:08.993870 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:30:08.995097 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:08.986568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46671 (local address 127.0.98.1:51909)
0929 16:30:08.987370 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:08.987411 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:08.988739 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:08.989339 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:08.989346 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:08.989435 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:08.989919 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:08.989931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.990747 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.990755 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:08.991646 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:08.991655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:08.992162 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:08.992167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:08.992333 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:08.993058 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:30:08.993061 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:08.993118 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:30:08.993253 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:08.993577 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:08.993587 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:08.993592 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 16:30:08.993778 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 16:30:08.994101 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":110,"threads_started":1}
I20260929 16:30:08.995814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46671
--builtin_ntp_servers=127.0.98.20:43649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:30:08.996338 19234 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46671
I20260929 16:30:08.996595 19234 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:08.997305 19234 heartbeater.cc:511] Master 127.0.98.62:46671 requested a full tablet report, sending...
I20260929 16:30:08.999055 19046 ts_manager.cc:194] Registered new tserver with Master: 2c49d0a3d222431fa732888acc20c6c5 (127.0.98.1:45907)
I20260929 16:30:09.000305 19046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:51909
W20260929 16:30:09.111974 19237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:09.112262 19237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:09.112330 19237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:09.116479 19237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:09.116570 19237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:09.116667 19237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:30:09.121650 19237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:09.122841 19237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:09.124239 19237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:09.134006 19243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:09.134006 19245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:09.134011 19242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:09.134451 19237 server_base.cc:1034] running on GCE node
I20260929 16:30:09.134900 19237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:09.135696 19237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:09.136919 19237 hybrid_clock.cc:648] HybridClock initialized: now 1790699409136885 us; error 52 us; skew 500 ppm
I20260929 16:30:09.139007 19237 webserver.cc:460] Webserver started at http://127.0.98.2:34621/ using document root <none> and password file <none>
I20260929 16:30:09.139621 19237 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:09.139717 19237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:09.139966 19237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:09.142453 19237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "6dd3fcaeaded4ecd91cd1103f0209203"
format_stamp: "Formatted at 2026-09-29 16:30:09 on dist-test-slave-vq9v"
server_key: "448d762849e62377c56dce0a6e2f734b"
server_key_iv: "8d47b9da278c753ab1e76a8f4cf4db20"
server_key_version: "encryptionkey@0"
I20260929 16:30:09.143086 19237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6dd3fcaeaded4ecd91cd1103f0209203"
format_stamp: "Formatted at 2026-09-29 16:30:09 on dist-test-slave-vq9v"
server_key: "448d762849e62377c56dce0a6e2f734b"
server_key_iv: "8d47b9da278c753ab1e76a8f4cf4db20"
server_key_version: "encryptionkey@0"
I20260929 16:30:09.147356 19237 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:30:09.150107 19251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:09.151415 19237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:09.151572 19237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "6dd3fcaeaded4ecd91cd1103f0209203"
format_stamp: "Formatted at 2026-09-29 16:30:09 on dist-test-slave-vq9v"
server_key: "448d762849e62377c56dce0a6e2f734b"
server_key_iv: "8d47b9da278c753ab1e76a8f4cf4db20"
server_key_version: "encryptionkey@0"
I20260929 16:30:09.151749 19237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:09.175271 19237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:09.176040 19237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:09.176257 19237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:09.176905 19237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:09.177966 19237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:09.178043 19237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:09.178115 19237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:09.178169 19237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:09.187456 19237 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40527
I20260929 16:30:09.187481 19364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40527 every 8 connection(s)
I20260929 16:30:09.188386 19237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:30:09.193435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 19237
I20260929 16:30:09.193591 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:30:09.193881 392 external_mini_cluster.cc:1428] Setting key 6ea75c0263cc095def47e42044055961
I20260929 16:30:09.195906 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46671
--builtin_ntp_servers=127.0.98.20:43649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:30:09.198378 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:09.190491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44831 (local address 127.0.98.62:46671)
0929 16:30:09.190635 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:09.190640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:09.191741 (+ 1101us) server_negotiation.cc:407] Connection header received
0929 16:30:09.193168 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:09.193172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:09.193260 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:09.193364 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:09.194414 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.195109 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.196340 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.196583 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.196695 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:09.197875 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:09.197879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:09.197881 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:09.197895 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:09.197923 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:09.197931 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:09.198011 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:09.198175 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 16:30:09.198237 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:30:09.199306 19367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:09.190862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46671 (local address 127.0.98.2:44831)
0929 16:30:09.191539 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:09.191582 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:09.192941 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:09.193559 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:09.193570 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:09.193682 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:09.194255 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:09.194269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.195235 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.195238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:09.196215 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:09.196225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.196755 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.196760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:09.196925 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:09.197593 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:30:09.197595 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:09.197636 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:30:09.197708 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:09.198013 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:09.198025 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:09.198033 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 16:30:09.198172 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 16:30:09.198555 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":116,"threads_started":1}
I20260929 16:30:09.200589 19365 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46671
I20260929 16:30:09.200879 19365 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:09.201457 19365 heartbeater.cc:511] Master 127.0.98.62:46671 requested a full tablet report, sending...
I20260929 16:30:09.202598 19046 ts_manager.cc:194] Registered new tserver with Master: 6dd3fcaeaded4ecd91cd1103f0209203 (127.0.98.2:40527)
I20260929 16:30:09.203473 19046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44831
W20260929 16:30:09.308974 19368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:09.309305 19368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:09.309347 19368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:09.313524 19368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:09.313584 19368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:09.313691 19368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:30:09.318709 19368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:09.319921 19368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:09.321422 19368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:09.331478 19373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:09.331598 19376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:09.331498 19374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:09.331998 19368 server_base.cc:1034] running on GCE node
I20260929 16:30:09.332459 19368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:09.333307 19368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:09.334544 19368 hybrid_clock.cc:648] HybridClock initialized: now 1790699409334521 us; error 42 us; skew 500 ppm
I20260929 16:30:09.336750 19368 webserver.cc:460] Webserver started at http://127.0.98.3:45981/ using document root <none> and password file <none>
I20260929 16:30:09.337464 19368 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:09.337532 19368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:09.337788 19368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:09.339933 19368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "de36386ec3384c94a0f027847d004faa"
format_stamp: "Formatted at 2026-09-29 16:30:09 on dist-test-slave-vq9v"
server_key: "1e4d4b1758b70e86859ee1987569c35c"
server_key_iv: "f01ee2fe0f33132b46411c528b2d549e"
server_key_version: "encryptionkey@0"
I20260929 16:30:09.340534 19368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "de36386ec3384c94a0f027847d004faa"
format_stamp: "Formatted at 2026-09-29 16:30:09 on dist-test-slave-vq9v"
server_key: "1e4d4b1758b70e86859ee1987569c35c"
server_key_iv: "f01ee2fe0f33132b46411c528b2d549e"
server_key_version: "encryptionkey@0"
I20260929 16:30:09.344588 19368 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:30:09.347365 19382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:09.348740 19368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:30:09.348842 19368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "de36386ec3384c94a0f027847d004faa"
format_stamp: "Formatted at 2026-09-29 16:30:09 on dist-test-slave-vq9v"
server_key: "1e4d4b1758b70e86859ee1987569c35c"
server_key_iv: "f01ee2fe0f33132b46411c528b2d549e"
server_key_version: "encryptionkey@0"
I20260929 16:30:09.349000 19368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:09.371626 19368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:09.372452 19368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:09.372649 19368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:09.373368 19368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:09.374341 19368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:09.374388 19368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:09.374429 19368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:09.374444 19368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:09.384595 19368 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35127
I20260929 16:30:09.384670 19495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35127 every 8 connection(s)
I20260929 16:30:09.385700 19368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:30:09.392896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 19368
I20260929 16:30:09.393026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:30:09.393373 392 external_mini_cluster.cc:1428] Setting key 3467613d729d24acafb4cbb25f43e976
I20260929 16:30:09.396252 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:09.387810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44281 (local address 127.0.98.62:46671)
0929 16:30:09.387967 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:09.387972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:09.389147 (+ 1175us) server_negotiation.cc:407] Connection header received
0929 16:30:09.390643 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:09.390646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:09.390740 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:09.390823 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:09.391773 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.392306 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.394078 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.394347 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.394469 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:09.395769 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:09.395776 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:09.395779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:09.395792 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:09.395814 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:09.395825 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:09.395889 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:09.396071 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 16:30:09.396130 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:30:09.397226 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:09.388281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46671 (local address 127.0.98.3:44281)
0929 16:30:09.388982 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:09.389026 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:09.390413 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:09.391000 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:09.391007 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:09.391099 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:09.391614 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:09.391627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.392460 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.392463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:09.393933 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:09.393944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.394512 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.394516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:09.394692 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:09.395408 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:30:09.395410 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:09.395454 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:30:09.395549 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:09.395908 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:09.395915 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:09.395919 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 16:30:09.396023 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 16:30:09.396275 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":111,"threads_started":1}
I20260929 16:30:09.398599 19496 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46671
I20260929 16:30:09.398892 19496 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:09.399459 19496 heartbeater.cc:511] Master 127.0.98.62:46671 requested a full tablet report, sending...
I20260929 16:30:09.400597 19046 ts_manager.cc:194] Registered new tserver with Master: de36386ec3384c94a0f027847d004faa (127.0.98.3:35127)
I20260929 16:30:09.401340 19046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:44281
I20260929 16:30:09.408080 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:30:09.414621 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:09.410296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57524 (local address 127.0.98.62:46671)
0929 16:30:09.410464 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:09.410470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:09.410631 (+ 161us) server_negotiation.cc:407] Connection header received
0929 16:30:09.410800 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:09.410803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:09.410860 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:09.410953 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:09.411264 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.411835 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.412738 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.412972 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.413090 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:09.414039 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:09.414042 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:09.414044 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:09.414057 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:09.414078 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:09.414086 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:09.414256 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:09.414419 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 16:30:09.414465 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:30:09.418123 19046 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:30:09.421159 19046 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:30:09.437695 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:09.430855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.0.98.2:40527)
0929 16:30:09.431497 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:09.431510 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:09.431942 (+ 432us) server_negotiation.cc:407] Connection header received
0929 16:30:09.432010 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:09.432014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:09.432467 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:09.432614 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:09.432908 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.434038 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.436174 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.436790 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.436927 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:09.437154 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:09.437380 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 16:30:09.437494 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":96,"threads_started":1}
I20260929 16:30:09.437677 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:09.430587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40527 (local address 127.0.0.1:47402)
0929 16:30:09.431688 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:09.431704 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:09.431810 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:09.432669 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:09.432674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:09.432693 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:09.432795 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:09.432804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.434161 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.434168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:09.436034 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:09.436045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.436942 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.436947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:09.437065 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:09.437107 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 16:30:09.437341 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 16:30:09.437494 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"mutex_wait_us":291,"thread_start_us":119,"threads_started":1}
I20260929 16:30:09.437939 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:09.430694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41500 (local address 127.0.98.1:45907)
0929 16:30:09.431309 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:09.431318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:09.431912 (+ 594us) server_negotiation.cc:407] Connection header received
0929 16:30:09.432005 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:09.432010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:09.432221 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:09.432386 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:09.432846 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.434117 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.436053 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.436868 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.436975 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:09.437155 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:09.437629 (+ 474us) server_negotiation.cc:299] Negotiation successful
0929 16:30:09.437749 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":115,"threads_started":1}
I20260929 16:30:09.437973 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:09.431463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35127 (local address 127.0.0.1:45558)
0929 16:30:09.432045 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:09.432058 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:09.432158 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:09.433187 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:09.433190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:09.433234 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:09.433323 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:09.433329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.434680 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.434686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:09.435929 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:09.435944 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.437246 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.437250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:09.437427 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:09.437487 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 16:30:09.437562 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 16:30:09.437642 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":125,"thread_start_us":100,"threads_started":1}
I20260929 16:30:09.438385 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:09.430583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45907 (local address 127.0.0.1:41500)
0929 16:30:09.431536 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:09.431597 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:09.431798 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:09.432403 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:09.432411 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:09.432435 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:09.432593 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:09.432602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.434217 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.434220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:09.435912 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:09.435924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.437246 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.437250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:09.437430 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:09.437493 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 16:30:09.438169 (+ 676us) client_negotiation.cc:240] Negotiation successful
0929 16:30:09.438259 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":109,"thread_start_us":129,"threads_started":1}
I20260929 16:30:09.438756 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:09.431623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45558 (local address 127.0.98.3:35127)
0929 16:30:09.432704 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:09.432718 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:09.432743 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:30:09.432842 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:09.432848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:09.433027 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:09.433188 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:09.433453 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.434573 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.436088 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.436868 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.436975 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:09.437167 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:09.437740 (+ 573us) server_negotiation.cc:299] Negotiation successful
0929 16:30:09.438420 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":966,"thread_start_us":99,"threads_started":1}
I20260929 16:30:09.441143 19299 tablet_service.cc:1467] Processing CreateTablet for tablet 9dc867cae469447bbb73ef6254c3f1cd (DEFAULT_TABLE table=test-table [id=e631a369c43244769e9bb263bb232a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:09.441135 19168 tablet_service.cc:1467] Processing CreateTablet for tablet 9dc867cae469447bbb73ef6254c3f1cd (DEFAULT_TABLE table=test-table [id=e631a369c43244769e9bb263bb232a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:09.442265 19299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dc867cae469447bbb73ef6254c3f1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:09.442286 19168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dc867cae469447bbb73ef6254c3f1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:09.443318 19430 tablet_service.cc:1467] Processing CreateTablet for tablet 9dc867cae469447bbb73ef6254c3f1cd (DEFAULT_TABLE table=test-table [id=e631a369c43244769e9bb263bb232a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:09.444428 19430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dc867cae469447bbb73ef6254c3f1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:09.449656 19513 tablet_bootstrap.cc:492] T 9dc867cae469447bbb73ef6254c3f1cd P 6dd3fcaeaded4ecd91cd1103f0209203: Bootstrap starting.
I20260929 16:30:09.450555 19514 tablet_bootstrap.cc:492] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5: Bootstrap starting.
I20260929 16:30:09.451499 19515 tablet_bootstrap.cc:492] T 9dc867cae469447bbb73ef6254c3f1cd P de36386ec3384c94a0f027847d004faa: Bootstrap starting.
I20260929 16:30:09.452826 19513 tablet_bootstrap.cc:654] T 9dc867cae469447bbb73ef6254c3f1cd P 6dd3fcaeaded4ecd91cd1103f0209203: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:09.452875 19514 tablet_bootstrap.cc:654] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:09.453562 19515 tablet_bootstrap.cc:654] T 9dc867cae469447bbb73ef6254c3f1cd P de36386ec3384c94a0f027847d004faa: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:09.453850 19513 log.cc:824] T 9dc867cae469447bbb73ef6254c3f1cd P 6dd3fcaeaded4ecd91cd1103f0209203: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:09.453850 19514 log.cc:824] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:09.454393 19515 log.cc:824] T 9dc867cae469447bbb73ef6254c3f1cd P de36386ec3384c94a0f027847d004faa: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:09.456233 19514 tablet_bootstrap.cc:492] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5: No bootstrap required, opened a new log
I20260929 16:30:09.456441 19514 ts_tablet_manager.cc:1397] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 16:30:09.456683 19515 tablet_bootstrap.cc:492] T 9dc867cae469447bbb73ef6254c3f1cd P de36386ec3384c94a0f027847d004faa: No bootstrap required, opened a new log
I20260929 16:30:09.456786 19513 tablet_bootstrap.cc:492] T 9dc867cae469447bbb73ef6254c3f1cd P 6dd3fcaeaded4ecd91cd1103f0209203: No bootstrap required, opened a new log
I20260929 16:30:09.456905 19515 ts_tablet_manager.cc:1397] T 9dc867cae469447bbb73ef6254c3f1cd P de36386ec3384c94a0f027847d004faa: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 16:30:09.456965 19513 ts_tablet_manager.cc:1397] T 9dc867cae469447bbb73ef6254c3f1cd P 6dd3fcaeaded4ecd91cd1103f0209203: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 16:30:09.460799 19515 raft_consensus.cc:348] T 9dc867cae469447bbb73ef6254c3f1cd P de36386ec3384c94a0f027847d004faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c49d0a3d222431fa732888acc20c6c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45907 } } peers { permanent_uuid: "6dd3fcaeaded4ecd91cd1103f0209203" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40527 } } peers { permanent_uuid: "de36386ec3384c94a0f027847d004faa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35127 } }
I20260929 16:30:09.461109 19515 raft_consensus.cc:374] T 9dc867cae469447bbb73ef6254c3f1cd P de36386ec3384c94a0f027847d004faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:09.461184 19515 raft_consensus.cc:729] T 9dc867cae469447bbb73ef6254c3f1cd P de36386ec3384c94a0f027847d004faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de36386ec3384c94a0f027847d004faa, State: Initialized, Role: FOLLOWER
I20260929 16:30:09.461292 19513 raft_consensus.cc:348] T 9dc867cae469447bbb73ef6254c3f1cd P 6dd3fcaeaded4ecd91cd1103f0209203 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c49d0a3d222431fa732888acc20c6c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45907 } } peers { permanent_uuid: "6dd3fcaeaded4ecd91cd1103f0209203" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40527 } } peers { permanent_uuid: "de36386ec3384c94a0f027847d004faa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35127 } }
I20260929 16:30:09.461239 19514 raft_consensus.cc:348] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c49d0a3d222431fa732888acc20c6c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45907 } } peers { permanent_uuid: "6dd3fcaeaded4ecd91cd1103f0209203" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40527 } } peers { permanent_uuid: "de36386ec3384c94a0f027847d004faa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35127 } }
I20260929 16:30:09.461491 19513 raft_consensus.cc:374] T 9dc867cae469447bbb73ef6254c3f1cd P 6dd3fcaeaded4ecd91cd1103f0209203 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:09.461491 19514 raft_consensus.cc:374] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:09.461539 19514 raft_consensus.cc:729] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c49d0a3d222431fa732888acc20c6c5, State: Initialized, Role: FOLLOWER
I20260929 16:30:09.461549 19513 raft_consensus.cc:729] T 9dc867cae469447bbb73ef6254c3f1cd P 6dd3fcaeaded4ecd91cd1103f0209203 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd3fcaeaded4ecd91cd1103f0209203, State: Initialized, Role: FOLLOWER
I20260929 16:30:09.461830 19515 consensus_queue.cc:260] T 9dc867cae469447bbb73ef6254c3f1cd P de36386ec3384c94a0f027847d004faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c49d0a3d222431fa732888acc20c6c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45907 } } peers { permanent_uuid: "6dd3fcaeaded4ecd91cd1103f0209203" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40527 } } peers { permanent_uuid: "de36386ec3384c94a0f027847d004faa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35127 } }
I20260929 16:30:09.461987 19514 consensus_queue.cc:260] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c49d0a3d222431fa732888acc20c6c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45907 } } peers { permanent_uuid: "6dd3fcaeaded4ecd91cd1103f0209203" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40527 } } peers { permanent_uuid: "de36386ec3384c94a0f027847d004faa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35127 } }
I20260929 16:30:09.461987 19513 consensus_queue.cc:260] T 9dc867cae469447bbb73ef6254c3f1cd P 6dd3fcaeaded4ecd91cd1103f0209203 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c49d0a3d222431fa732888acc20c6c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45907 } } peers { permanent_uuid: "6dd3fcaeaded4ecd91cd1103f0209203" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40527 } } peers { permanent_uuid: "de36386ec3384c94a0f027847d004faa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35127 } }
I20260929 16:30:09.462992 19513 ts_tablet_manager.cc:1428] T 9dc867cae469447bbb73ef6254c3f1cd P 6dd3fcaeaded4ecd91cd1103f0209203: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:30:09.462992 19514 ts_tablet_manager.cc:1428] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 16:30:09.462999 19515 ts_tablet_manager.cc:1428] T 9dc867cae469447bbb73ef6254c3f1cd P de36386ec3384c94a0f027847d004faa: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260929 16:30:09.463366 19365 heartbeater.cc:503] Master 127.0.98.62:46671 was elected leader, sending a full tablet report...
I20260929 16:30:09.463366 19496 heartbeater.cc:503] Master 127.0.98.62:46671 was elected leader, sending a full tablet report...
I20260929 16:30:09.464452 19234 heartbeater.cc:503] Master 127.0.98.62:46671 was elected leader, sending a full tablet report...
W20260929 16:30:09.485327 19235 tablet.cc:2369] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:30:09.636807 19497 tablet.cc:2369] T 9dc867cae469447bbb73ef6254c3f1cd P de36386ec3384c94a0f027847d004faa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:30:09.689589 19366 tablet.cc:2369] T 9dc867cae469447bbb73ef6254c3f1cd P 6dd3fcaeaded4ecd91cd1103f0209203: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:30:09.811558 19521 raft_consensus.cc:482] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:30:09.811853 19521 raft_consensus.cc:504] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c49d0a3d222431fa732888acc20c6c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45907 } } peers { permanent_uuid: "6dd3fcaeaded4ecd91cd1103f0209203" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40527 } } peers { permanent_uuid: "de36386ec3384c94a0f027847d004faa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35127 } }
I20260929 16:30:09.813072 19521 leader_election.cc:288] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dd3fcaeaded4ecd91cd1103f0209203 (127.0.98.2:40527), de36386ec3384c94a0f027847d004faa (127.0.98.3:35127)
I20260929 16:30:09.817369 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:09.813388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40527 (local address 127.0.98.1:37157)
0929 16:30:09.813758 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:09.813774 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:09.813930 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:09.814207 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:09.814210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:09.814226 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:09.814392 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:09.814398 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.815191 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.815196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:09.816190 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:09.816201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.816846 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.816850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:09.816975 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:09.817009 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:30:09.817094 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 16:30:09.817182 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":32,"spinlock_wait_cycles":26880,"thread_start_us":122,"threads_started":1}
I20260929 16:30:09.817348 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:09.813578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37157 (local address 127.0.98.2:40527)
0929 16:30:09.813744 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:09.813749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:09.813867 (+ 118us) server_negotiation.cc:407] Connection header received
0929 16:30:09.814041 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:09.814044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:09.814097 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:09.814186 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:09.814519 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.815068 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.816327 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.816718 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.816839 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:09.816901 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:09.817125 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 16:30:09.817221 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:30:09.817431 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:09.813507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40355 (local address 127.0.98.3:35127)
0929 16:30:09.813647 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:09.813654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:09.813925 (+ 271us) server_negotiation.cc:407] Connection header received
0929 16:30:09.814094 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:09.814096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:09.814136 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:09.814230 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:09.814536 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.815084 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.816334 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.816746 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.816867 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:09.816949 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:09.817237 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 16:30:09.817314 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:30:09.817484 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:09.813371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35127 (local address 127.0.98.1:40355)
0929 16:30:09.813817 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:09.813829 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:09.813963 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:09.814239 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:09.814242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:09.814279 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:09.814419 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:09.814424 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.815200 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.815204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:09.816208 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:09.816218 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.816870 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.816873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:09.817019 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:09.817046 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:30:09.817215 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 16:30:09.817292 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":86,"spinlock_wait_cycles":768,"thread_start_us":108,"threads_started":1}
I20260929 16:30:09.818289 19319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc867cae469447bbb73ef6254c3f1cd" candidate_uuid: "2c49d0a3d222431fa732888acc20c6c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dd3fcaeaded4ecd91cd1103f0209203" is_pre_election: true
I20260929 16:30:09.818300 19450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc867cae469447bbb73ef6254c3f1cd" candidate_uuid: "2c49d0a3d222431fa732888acc20c6c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de36386ec3384c94a0f027847d004faa" is_pre_election: true
I20260929 16:30:09.818642 19319 raft_consensus.cc:2454] T 9dc867cae469447bbb73ef6254c3f1cd P 6dd3fcaeaded4ecd91cd1103f0209203 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c49d0a3d222431fa732888acc20c6c5 in term 0.
I20260929 16:30:09.818650 19450 raft_consensus.cc:2454] T 9dc867cae469447bbb73ef6254c3f1cd P de36386ec3384c94a0f027847d004faa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c49d0a3d222431fa732888acc20c6c5 in term 0.
I20260929 16:30:09.819276 19123 leader_election.cc:302] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c49d0a3d222431fa732888acc20c6c5, de36386ec3384c94a0f027847d004faa; no voters:
I20260929 16:30:09.819552 19521 raft_consensus.cc:2781] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:30:09.819706 19521 raft_consensus.cc:482] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:30:09.819777 19521 raft_consensus.cc:3037] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:09.820880 19521 raft_consensus.cc:504] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c49d0a3d222431fa732888acc20c6c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45907 } } peers { permanent_uuid: "6dd3fcaeaded4ecd91cd1103f0209203" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40527 } } peers { permanent_uuid: "de36386ec3384c94a0f027847d004faa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35127 } }
I20260929 16:30:09.821442 19521 leader_election.cc:288] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5 [CANDIDATE]: Term 1 election: Requested vote from peers 6dd3fcaeaded4ecd91cd1103f0209203 (127.0.98.2:40527), de36386ec3384c94a0f027847d004faa (127.0.98.3:35127)
I20260929 16:30:09.821858 19450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc867cae469447bbb73ef6254c3f1cd" candidate_uuid: "2c49d0a3d222431fa732888acc20c6c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de36386ec3384c94a0f027847d004faa"
I20260929 16:30:09.821923 19319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc867cae469447bbb73ef6254c3f1cd" candidate_uuid: "2c49d0a3d222431fa732888acc20c6c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dd3fcaeaded4ecd91cd1103f0209203"
I20260929 16:30:09.822011 19450 raft_consensus.cc:3037] T 9dc867cae469447bbb73ef6254c3f1cd P de36386ec3384c94a0f027847d004faa [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:09.822077 19319 raft_consensus.cc:3037] T 9dc867cae469447bbb73ef6254c3f1cd P 6dd3fcaeaded4ecd91cd1103f0209203 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:09.823119 19450 raft_consensus.cc:2454] T 9dc867cae469447bbb73ef6254c3f1cd P de36386ec3384c94a0f027847d004faa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c49d0a3d222431fa732888acc20c6c5 in term 1.
I20260929 16:30:09.823136 19319 raft_consensus.cc:2454] T 9dc867cae469447bbb73ef6254c3f1cd P 6dd3fcaeaded4ecd91cd1103f0209203 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c49d0a3d222431fa732888acc20c6c5 in term 1.
I20260929 16:30:09.823503 19123 leader_election.cc:302] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c49d0a3d222431fa732888acc20c6c5, de36386ec3384c94a0f027847d004faa; no voters:
I20260929 16:30:09.823705 19521 raft_consensus.cc:2781] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:09.823942 19521 raft_consensus.cc:686] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5 [term 1 LEADER]: Becoming Leader. State: Replica: 2c49d0a3d222431fa732888acc20c6c5, State: Running, Role: LEADER
I20260929 16:30:09.824338 19521 consensus_queue.cc:237] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c49d0a3d222431fa732888acc20c6c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45907 } } peers { permanent_uuid: "6dd3fcaeaded4ecd91cd1103f0209203" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40527 } } peers { permanent_uuid: "de36386ec3384c94a0f027847d004faa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35127 } }
I20260929 16:30:09.827759 19045 catalog_manager.cc:5495] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c49d0a3d222431fa732888acc20c6c5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2c49d0a3d222431fa732888acc20c6c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c49d0a3d222431fa732888acc20c6c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dd3fcaeaded4ecd91cd1103f0209203" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de36386ec3384c94a0f027847d004faa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35127 } health_report { overall_health: UNKNOWN } } }
I20260929 16:30:09.895442 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:09.891592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41502 (local address 127.0.98.1:45907)
0929 16:30:09.891810 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:09.891815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:09.891831 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:30:09.891914 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:09.891916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:09.891971 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:09.892099 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:30:09.892377 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.892900 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.893845 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:09.894068 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:09.894182 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:09.894540 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:30:09.894973 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:30:09.895137 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:09.895247 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 16:30:09.895286 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260929 16:30:09.901584 19319 raft_consensus.cc:1261] T 9dc867cae469447bbb73ef6254c3f1cd P 6dd3fcaeaded4ecd91cd1103f0209203 [term 1 FOLLOWER]: Refusing update from remote peer 2c49d0a3d222431fa732888acc20c6c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:09.901584 19450 raft_consensus.cc:1261] T 9dc867cae469447bbb73ef6254c3f1cd P de36386ec3384c94a0f027847d004faa [term 1 FOLLOWER]: Refusing update from remote peer 2c49d0a3d222431fa732888acc20c6c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:09.902465 19521 consensus_queue.cc:1035] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de36386ec3384c94a0f027847d004faa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:09.902698 19524 consensus_queue.cc:1035] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dd3fcaeaded4ecd91cd1103f0209203" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:09.911095 19527 mvcc.cc:205] Tried to move back new op lower bound from 7334704782948999168 to 7334704782643412992. Current Snapshot: MvccSnapshot[applied={T|T < 7334704782948999168}]
I20260929 16:30:09.913131 19529 mvcc.cc:205] Tried to move back new op lower bound from 7334704782948999168 to 7334704782643412992. Current Snapshot: MvccSnapshot[applied={T|T < 7334704782948999168 or (T in {7334704782948999168})}]
I20260929 16:30:09.913367 19528 mvcc.cc:205] Tried to move back new op lower bound from 7334704782948999168 to 7334704782643412992. Current Snapshot: MvccSnapshot[applied={T|T < 7334704782948999168 or (T in {7334704782948999168})}]
I20260929 16:30:09.921715 19045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:30:09.921897 19045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:30:09.924844 19045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2314f19bfe0049b99da8f2ee103b1c13: Sending DeleteTablet for 3 replicas of tablet 9dc867cae469447bbb73ef6254c3f1cd
I20260929 16:30:09.925879 19299 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dc867cae469447bbb73ef6254c3f1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:09 UTC) from {username='slave'} at 127.0.0.1:47402
I20260929 16:30:09.925927 19168 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dc867cae469447bbb73ef6254c3f1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:09 UTC) from {username='slave'} at 127.0.0.1:41500
I20260929 16:30:09.925971 19430 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dc867cae469447bbb73ef6254c3f1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:09 UTC) from {username='slave'} at 127.0.0.1:45558
I20260929 16:30:09.926525 19538 tablet_replica.cc:330] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5: stopping tablet replica
I20260929 16:30:09.926717 19537 tablet_replica.cc:330] T 9dc867cae469447bbb73ef6254c3f1cd P 6dd3fcaeaded4ecd91cd1103f0209203: stopping tablet replica
I20260929 16:30:09.926926 19538 raft_consensus.cc:2229] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:30:09.927147 19537 raft_consensus.cc:2229] T 9dc867cae469447bbb73ef6254c3f1cd P 6dd3fcaeaded4ecd91cd1103f0209203 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:30:09.927433 19538 raft_consensus.cc:2258] T 9dc867cae469447bbb73ef6254c3f1cd P 2c49d0a3d222431fa732888acc20c6c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:09.927493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 19106
I20260929 16:30:09.927614 19537 raft_consensus.cc:2258] T 9dc867cae469447bbb73ef6254c3f1cd P 6dd3fcaeaded4ecd91cd1103f0209203 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:09.928357 19539 tablet_replica.cc:330] T 9dc867cae469447bbb73ef6254c3f1cd P de36386ec3384c94a0f027847d004faa: stopping tablet replica
I20260929 16:30:09.928678 19539 raft_consensus.cc:2229] T 9dc867cae469447bbb73ef6254c3f1cd P de36386ec3384c94a0f027847d004faa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:30:09.929003 19539 raft_consensus.cc:2258] T 9dc867cae469447bbb73ef6254c3f1cd P de36386ec3384c94a0f027847d004faa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:09.930505 19537 ts_tablet_manager.cc:1905] T 9dc867cae469447bbb73ef6254c3f1cd P 6dd3fcaeaded4ecd91cd1103f0209203: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:30:09.930776 19539 ts_tablet_manager.cc:1905] T 9dc867cae469447bbb73ef6254c3f1cd P de36386ec3384c94a0f027847d004faa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:30:09.933337 19537 ts_tablet_manager.cc:1918] T 9dc867cae469447bbb73ef6254c3f1cd P 6dd3fcaeaded4ecd91cd1103f0209203: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:30:09.933337 19539 ts_tablet_manager.cc:1918] T 9dc867cae469447bbb73ef6254c3f1cd P de36386ec3384c94a0f027847d004faa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:30:09.933439 19537 log.cc:1196] T 9dc867cae469447bbb73ef6254c3f1cd P 6dd3fcaeaded4ecd91cd1103f0209203: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/9dc867cae469447bbb73ef6254c3f1cd
I20260929 16:30:09.933439 19539 log.cc:1196] T 9dc867cae469447bbb73ef6254c3f1cd P de36386ec3384c94a0f027847d004faa: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/9dc867cae469447bbb73ef6254c3f1cd
I20260929 16:30:09.933765 19539 ts_tablet_manager.cc:1939] T 9dc867cae469447bbb73ef6254c3f1cd P de36386ec3384c94a0f027847d004faa: Deleting consensus metadata
I20260929 16:30:09.933765 19537 ts_tablet_manager.cc:1939] T 9dc867cae469447bbb73ef6254c3f1cd P 6dd3fcaeaded4ecd91cd1103f0209203: Deleting consensus metadata
I20260929 16:30:09.935233 19030 catalog_manager.cc:4841] TS 6dd3fcaeaded4ecd91cd1103f0209203 (127.0.98.2:40527): tablet 9dc867cae469447bbb73ef6254c3f1cd (table test-table [id=e631a369c43244769e9bb263bb232a44]) successfully deleted
I20260929 16:30:09.935250 19032 catalog_manager.cc:4841] TS de36386ec3384c94a0f027847d004faa (127.0.98.3:35127): tablet 9dc867cae469447bbb73ef6254c3f1cd (table test-table [id=e631a369c43244769e9bb263bb232a44]) successfully deleted
W20260929 16:30:09.936182 19032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45907 (error 108)
I20260929 16:30:09.936368 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 19237
I20260929 16:30:09.937953 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:09.937616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45907 (local address 127.0.0.1:41508)
0929 16:30:09.937753 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:09.937825 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 16:30:09.938130 19032 catalog_manager.cc:4584] TS 2c49d0a3d222431fa732888acc20c6c5 (127.0.98.1:45907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9dc867cae469447bbb73ef6254c3f1cd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45907: connect: Connection refused (error 111)
I20260929 16:30:09.943266 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 19368
I20260929 16:30:09.948932 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 19015
2026-09-29T16:30:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1451 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:10 dist-test-slave-vq9v krb5kdc[19543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:10 dist-test-slave-vq9v krb5kdc[19543](info): set up 2 sockets
Sep 29 16:30:10 dist-test-slave-vq9v krb5kdc[19543](info): commencing operation
krb5kdc: starting...
W20260929 16:30:10.994582 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:11 dist-test-slave-vq9v krb5kdc[19543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:30:11.132157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36877
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:11.244434 19556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:11.244833 19556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:11.244922 19556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:11.249066 19556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:11.249181 19556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:11.249253 19556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:11.249294 19556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:11.249336 19556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:11.254853 19556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:11.256026 19556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:11.257140 19556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:11.265763 19561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:11.265779 19562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:11.265846 19556 server_base.cc:1034] running on GCE node
W20260929 16:30:11.265790 19564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:11.266657 19556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:11.267846 19556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:11.269043 19556 hybrid_clock.cc:648] HybridClock initialized: now 1790699411269014 us; error 44 us; skew 500 ppm
Sep 29 16:30:11 dist-test-slave-vq9v krb5kdc[19543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:11.271999 19556 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:30:11.272966 19556 webserver.cc:460] Webserver started at http://127.0.98.62:42141/ using document root <none> and password file <none>
I20260929 16:30:11.273613 19556 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:11.273708 19556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:11.273957 19556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:11.276072 19556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "d8c0558821164f4186ed492026f8143f"
format_stamp: "Formatted at 2026-09-29 16:30:11 on dist-test-slave-vq9v"
server_key: "4dc93a9ed452cd4709570f42fe89fc37"
server_key_iv: "2cb5eb215e76b4f37400025cbb9f0c2f"
server_key_version: "encryptionkey@0"
I20260929 16:30:11.276664 19556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "d8c0558821164f4186ed492026f8143f"
format_stamp: "Formatted at 2026-09-29 16:30:11 on dist-test-slave-vq9v"
server_key: "4dc93a9ed452cd4709570f42fe89fc37"
server_key_iv: "2cb5eb215e76b4f37400025cbb9f0c2f"
server_key_version: "encryptionkey@0"
I20260929 16:30:11.280733 19556 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:30:11.283470 19571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:11.285022 19556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:30:11.285252 19556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "d8c0558821164f4186ed492026f8143f"
format_stamp: "Formatted at 2026-09-29 16:30:11 on dist-test-slave-vq9v"
server_key: "4dc93a9ed452cd4709570f42fe89fc37"
server_key_iv: "2cb5eb215e76b4f37400025cbb9f0c2f"
server_key_version: "encryptionkey@0"
I20260929 16:30:11.285422 19556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:11.318089 19556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:11.321923 19556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:11.322190 19556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:11.330178 19556 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36877
I20260929 16:30:11.330207 19623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36877 every 8 connection(s)
I20260929 16:30:11.331249 19556 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:11.334246 19624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:11.339337 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 19556
I20260929 16:30:11.339465 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:11.339821 392 external_mini_cluster.cc:1428] Setting key 67e310b4fe78e76d237d2568d4a3d61d
I20260929 16:30:11.339973 19624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f: Bootstrap starting.
I20260929 16:30:11.342860 19624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:11.343787 19624 log.cc:824] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:11.346186 19624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f: No bootstrap required, opened a new log
Sep 29 16:30:11 dist-test-slave-vq9v krb5kdc[19543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:30:11.351346 19624 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c0558821164f4186ed492026f8143f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36877 } }
I20260929 16:30:11.351722 19624 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:11.351830 19624 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c0558821164f4186ed492026f8143f, State: Initialized, Role: FOLLOWER
I20260929 16:30:11.352383 19624 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c0558821164f4186ed492026f8143f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36877 } }
I20260929 16:30:11.352552 19624 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:11.352638 19624 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:11.352758 19624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:11.353888 19624 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c0558821164f4186ed492026f8143f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36877 } }
I20260929 16:30:11.354282 19624 leader_election.cc:302] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8c0558821164f4186ed492026f8143f; no voters:
I20260929 16:30:11.354725 19624 leader_election.cc:288] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:11.354806 19629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:11.355036 19629 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f [term 1 LEADER]: Becoming Leader. State: Replica: d8c0558821164f4186ed492026f8143f, State: Running, Role: LEADER
I20260929 16:30:11.355439 19629 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c0558821164f4186ed492026f8143f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36877 } }
I20260929 16:30:11.355756 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:11.341339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.0.98.62:36877)
0929 16:30:11.341792 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:11.341812 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:11.341855 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:30:11.342691 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:11.342718 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:11.343178 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:11.343566 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:11.344459 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:11.345583 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:11.346854 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:11.347333 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:11.347467 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:11.350418 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:11.350423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:11.350439 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:11.350467 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:11.352411 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:11.352964 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:11.352970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:11.352974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:11.353090 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:11.353419 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:11.353423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:11.353425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:11.353841 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:11.354044 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:11.354370 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 16:30:11.354721 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":136,"threads_started":1}
I20260929 16:30:11.356142 19624 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:11.357165 19631 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8c0558821164f4186ed492026f8143f. Latest consensus state: current_term: 1 leader_uuid: "d8c0558821164f4186ed492026f8143f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c0558821164f4186ed492026f8143f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36877 } } }
I20260929 16:30:11.357381 19631 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:11.357399 19630 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8c0558821164f4186ed492026f8143f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c0558821164f4186ed492026f8143f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36877 } } }
I20260929 16:30:11.357478 19630 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:11.358135 19638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:11.361274 19638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:11.367606 19638 catalog_manager.cc:1347] Generated new cluster ID: bc758e67874c4247a3dffb69a2c7708f
I20260929 16:30:11.367692 19638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:11.410393 19638 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:11.411862 19638 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:11.423812 19638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f: Generated new TSK 0
I20260929 16:30:11.424736 19638 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 16:30:11.465629 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36877
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:11.604203 19650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:11.604485 19650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:11.604530 19650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:11.608605 19650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:11.608680 19650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:11.608743 19650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:30:11.614346 19650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:11.615430 19650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:11.616428 19650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:11.626607 19655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:11.626693 19658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:11.626607 19656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:11.626883 19650 server_base.cc:1034] running on GCE node
I20260929 16:30:11.627542 19650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:11.628235 19650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:11.629426 19650 hybrid_clock.cc:648] HybridClock initialized: now 1790699411629387 us; error 62 us; skew 500 ppm
Sep 29 16:30:11 dist-test-slave-vq9v krb5kdc[19543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:11.632323 19650 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 16:30:11.633380 19650 webserver.cc:460] Webserver started at http://127.0.98.1:46333/ using document root <none> and password file <none>
I20260929 16:30:11.633992 19650 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:11.634089 19650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:11.634347 19650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:11.636511 19650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "7f8a620dfd4c4941b4b0ac88b679861d"
format_stamp: "Formatted at 2026-09-29 16:30:11 on dist-test-slave-vq9v"
server_key: "6cc389877105e13868f8f17d42308e7f"
server_key_iv: "fdedfb4be1d633e3bc4f6953aad1b665"
server_key_version: "encryptionkey@0"
I20260929 16:30:11.637141 19650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7f8a620dfd4c4941b4b0ac88b679861d"
format_stamp: "Formatted at 2026-09-29 16:30:11 on dist-test-slave-vq9v"
server_key: "6cc389877105e13868f8f17d42308e7f"
server_key_iv: "fdedfb4be1d633e3bc4f6953aad1b665"
server_key_version: "encryptionkey@0"
I20260929 16:30:11.641534 19650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:30:11.644428 19665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:11.646010 19650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:11.646171 19650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "7f8a620dfd4c4941b4b0ac88b679861d"
format_stamp: "Formatted at 2026-09-29 16:30:11 on dist-test-slave-vq9v"
server_key: "6cc389877105e13868f8f17d42308e7f"
server_key_iv: "fdedfb4be1d633e3bc4f6953aad1b665"
server_key_version: "encryptionkey@0"
I20260929 16:30:11.646328 19650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:11.658278 19650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:11.661367 19650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:11.661574 19650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:11.662173 19650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:11.663141 19650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:11.663214 19650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:11.663313 19650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:11.663367 19650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:11.673102 19650 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42629
I20260929 16:30:11.673136 19778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42629 every 8 connection(s)
I20260929 16:30:11.674072 19650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:30:11.684052 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 19650
I20260929 16:30:11.684204 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
Sep 29 16:30:11 dist-test-slave-vq9v krb5kdc[19543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:30:11.684556 392 external_mini_cluster.cc:1428] Setting key 46e9a3ad5b2fcb1242d2db57681aa455
I20260929 16:30:11.689304 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:11.676003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34939 (local address 127.0.98.62:36877)
0929 16:30:11.676161 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:11.676167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:11.677091 (+ 924us) server_negotiation.cc:407] Connection header received
0929 16:30:11.678516 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:11.678519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:11.678610 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:11.678705 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:11.680038 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:11.680625 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:11.681981 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:11.682226 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:11.682322 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:11.685273 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:11.685276 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:11.685278 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:11.685303 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:11.686788 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:11.687464 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:11.687467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:11.687468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:11.687542 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:11.687967 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:11.687972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:11.687975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:11.688245 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:11.688355 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:11.688918 (+ 563us) server_negotiation.cc:299] Negotiation successful
0929 16:30:11.689071 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:30:11.690440 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:11.676462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36877 (local address 127.0.98.1:34939)
0929 16:30:11.676921 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:11.676966 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:11.678284 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:11.678893 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:11.678900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:11.679355 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:11.679857 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:11.679871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:11.680784 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:11.680787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:11.681852 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:11.681862 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:11.682392 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:11.682395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:11.682542 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:11.683170 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:11.683187 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:11.685025 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:11.686914 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:11.686924 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:11.686946 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:11.687360 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:11.687631 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:11.687634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:11.687636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:11.687857 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:11.688389 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:11.688398 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:11.688668 (+ 270us) client_negotiation.cc:769] Sending connection context
0929 16:30:11.688832 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 16:30:11.689258 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":122,"threads_started":1}
I20260929 16:30:11.691782 19779 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36877
I20260929 16:30:11.692047 19779 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:11.692572 19779 heartbeater.cc:511] Master 127.0.98.62:36877 requested a full tablet report, sending...
I20260929 16:30:11.694149 19588 ts_manager.cc:194] Registered new tserver with Master: 7f8a620dfd4c4941b4b0ac88b679861d (127.0.98.1:42629)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 16:30:11.695369 19588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 16:30:11.716069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36877
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:11.831238 19784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:11.831619 19784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:11.831727 19784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:11.836105 19784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:11.836221 19784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:11.836325 19784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:30:11.842036 19784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:11.843190 19784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:11.844262 19784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:11.854130 19792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:11.854128 19790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:11.854161 19789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:11.854512 19784 server_base.cc:1034] running on GCE node
I20260929 16:30:11.855031 19784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:11.855731 19784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:11.856899 19784 hybrid_clock.cc:648] HybridClock initialized: now 1790699411856861 us; error 50 us; skew 500 ppm
Sep 29 16:30:11 dist-test-slave-vq9v krb5kdc[19543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:11.859820 19784 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 16:30:11.860725 19784 webserver.cc:460] Webserver started at http://127.0.98.2:40979/ using document root <none> and password file <none>
I20260929 16:30:11.861390 19784 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:11.861490 19784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:11.861732 19784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:11.864039 19784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "0948f6387e8042e99a883b0bbf352341"
format_stamp: "Formatted at 2026-09-29 16:30:11 on dist-test-slave-vq9v"
server_key: "1d19b43dc565f9e64c6c10db998f3163"
server_key_iv: "92531792ed80316dc7a2c127bd8f9334"
server_key_version: "encryptionkey@0"
I20260929 16:30:11.864640 19784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0948f6387e8042e99a883b0bbf352341"
format_stamp: "Formatted at 2026-09-29 16:30:11 on dist-test-slave-vq9v"
server_key: "1d19b43dc565f9e64c6c10db998f3163"
server_key_iv: "92531792ed80316dc7a2c127bd8f9334"
server_key_version: "encryptionkey@0"
I20260929 16:30:11.868625 19784 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:30:11.871309 19799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:11.872777 19784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:30:11.872915 19784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "0948f6387e8042e99a883b0bbf352341"
format_stamp: "Formatted at 2026-09-29 16:30:11 on dist-test-slave-vq9v"
server_key: "1d19b43dc565f9e64c6c10db998f3163"
server_key_iv: "92531792ed80316dc7a2c127bd8f9334"
server_key_version: "encryptionkey@0"
I20260929 16:30:11.873077 19784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:11.891494 19784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:11.895262 19784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:11.895517 19784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:11.896188 19784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:11.897096 19784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:11.897147 19784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:11.897192 19784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:11.897240 19784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:11.907225 19784 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36089
I20260929 16:30:11.907244 19912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36089 every 8 connection(s)
I20260929 16:30:11.908277 19784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:30:11.913447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 19784
I20260929 16:30:11.913616 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:30:11.913946 392 external_mini_cluster.cc:1428] Setting key 37339e17ef4fd3cc66463af1b3a51b49
Sep 29 16:30:11 dist-test-slave-vq9v krb5kdc[19543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 16:30:11.925944 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:11.910381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58163 (local address 127.0.98.62:36877)
0929 16:30:11.910552 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:11.910556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:11.911533 (+ 977us) server_negotiation.cc:407] Connection header received
0929 16:30:11.912967 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:11.912970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:11.913023 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:11.913180 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:11.914749 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:11.915282 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:11.917429 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:11.917833 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:11.917965 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:11.921127 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:11.921130 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:11.921132 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:11.921158 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:11.922930 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:11.923606 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:11.923610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:11.923612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:11.923734 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:11.924315 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:11.924322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:11.924327 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:11.924641 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:11.924791 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:11.925618 (+ 827us) server_negotiation.cc:299] Negotiation successful
0929 16:30:11.925806 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:30:11.926993 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:11.910822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36877 (local address 127.0.98.2:58163)
0929 16:30:11.911354 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:11.911405 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:11.912708 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:11.913404 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:11.913412 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:11.913905 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:11.914595 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:11.914616 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:11.915462 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:11.915465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:11.917174 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:11.917191 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:11.918026 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:11.918029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:11.918207 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:11.918974 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:11.919003 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:11.920905 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:11.923103 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:11.923111 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:11.923130 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:11.923490 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:11.923944 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:11.923948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:11.923950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:11.924186 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:11.924785 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:11.924792 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:11.925191 (+ 399us) client_negotiation.cc:769] Sending connection context
0929 16:30:11.925511 (+ 320us) client_negotiation.cc:240] Negotiation successful
0929 16:30:11.925942 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":149,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 16:30:11.928499 19913 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36877
I20260929 16:30:11.928826 19913 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:11.929523 19913 heartbeater.cc:511] Master 127.0.98.62:36877 requested a full tablet report, sending...
I20260929 16:30:11.930924 19588 ts_manager.cc:194] Registered new tserver with Master: 0948f6387e8042e99a883b0bbf352341 (127.0.98.2:36089)
I20260929 16:30:11.931849 19588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 16:30:11.945761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36877
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:12.065974 19918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:12.066291 19918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:12.066352 19918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:12.071074 19918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:12.071178 19918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:12.071244 19918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:30:12.077157 19918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:12.078341 19918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:12.079351 19918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:12.090142 19923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:12.090143 19924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:12.090186 19918 server_base.cc:1034] running on GCE node
W20260929 16:30:12.090142 19926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:12.090880 19918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:12.091624 19918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:12.092831 19918 hybrid_clock.cc:648] HybridClock initialized: now 1790699412092801 us; error 41 us; skew 500 ppm
Sep 29 16:30:12 dist-test-slave-vq9v krb5kdc[19543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:12.095731 19918 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 16:30:12.096633 19918 webserver.cc:460] Webserver started at http://127.0.98.3:34473/ using document root <none> and password file <none>
I20260929 16:30:12.097282 19918 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:12.097378 19918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:12.097615 19918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:12.099757 19918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "ad3207ed46f64b50acd38fab19f80535"
format_stamp: "Formatted at 2026-09-29 16:30:12 on dist-test-slave-vq9v"
server_key: "b2edc535b864683bfbae48b36e26434b"
server_key_iv: "ebc4e1b25acf93465cb7e8fb20890bd6"
server_key_version: "encryptionkey@0"
I20260929 16:30:12.100356 19918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ad3207ed46f64b50acd38fab19f80535"
format_stamp: "Formatted at 2026-09-29 16:30:12 on dist-test-slave-vq9v"
server_key: "b2edc535b864683bfbae48b36e26434b"
server_key_iv: "ebc4e1b25acf93465cb7e8fb20890bd6"
server_key_version: "encryptionkey@0"
I20260929 16:30:12.104314 19918 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 16:30:12.106875 19933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:12.108110 19918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:30:12.108253 19918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "ad3207ed46f64b50acd38fab19f80535"
format_stamp: "Formatted at 2026-09-29 16:30:12 on dist-test-slave-vq9v"
server_key: "b2edc535b864683bfbae48b36e26434b"
server_key_iv: "ebc4e1b25acf93465cb7e8fb20890bd6"
server_key_version: "encryptionkey@0"
I20260929 16:30:12.108372 19918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:12.137454 19918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:12.141010 19918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:12.141299 19918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:12.141928 19918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:12.142828 19918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:12.142899 19918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:12.142966 19918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:12.143020 19918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:12.152813 20046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36541 every 8 connection(s)
I20260929 16:30:12.152808 19918 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36541
I20260929 16:30:12.153918 19918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:30:12.163244 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 19918
I20260929 16:30:12.163403 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:30:12.163774 392 external_mini_cluster.cc:1428] Setting key 98c7ef1f924e4211d1846299440c6961
Sep 29 16:30:12 dist-test-slave-vq9v krb5kdc[19543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:30:12.170495 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:12.155884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56265 (local address 127.0.98.62:36877)
0929 16:30:12.156057 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:12.156062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:12.157046 (+ 984us) server_negotiation.cc:407] Connection header received
0929 16:30:12.158570 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:12.158574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:12.158641 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:12.158718 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:12.160039 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.160612 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.161827 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.162039 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.162127 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:12.165685 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:12.165690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:12.165694 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:12.165740 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:12.167568 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:12.168250 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:12.168253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:12.168255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:12.168370 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:12.168863 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:12.168866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:12.168868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:12.169153 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:12.169316 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:12.170069 (+ 753us) server_negotiation.cc:299] Negotiation successful
0929 16:30:12.170271 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:30:12.171643 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:12.156320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36877 (local address 127.0.98.3:56265)
0929 16:30:12.156826 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:12.156897 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:12.158308 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:12.158888 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:12.158895 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:12.159379 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:12.159878 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:12.159891 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.160727 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.160730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:12.161698 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:12.161708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.162179 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.162182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:12.162408 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:12.163219 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:12.163244 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:12.165411 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:12.167709 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:12.167716 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:12.167734 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:12.168126 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:12.168479 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:12.168482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:12.168484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:12.168730 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:12.169334 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:12.169341 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:12.169750 (+ 409us) client_negotiation.cc:769] Sending connection context
0929 16:30:12.170003 (+ 253us) client_negotiation.cc:240] Negotiation successful
0929 16:30:12.170454 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":126,"threads_started":1}
I20260929 16:30:12.172900 20047 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36877
I20260929 16:30:12.173177 20047 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:12.173743 20047 heartbeater.cc:511] Master 127.0.98.62:36877 requested a full tablet report, sending...
I20260929 16:30:12.174881 19588 ts_manager.cc:194] Registered new tserver with Master: ad3207ed46f64b50acd38fab19f80535 (127.0.98.3:36541)
I20260929 16:30:12.175513 19588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56265
I20260929 16:30:12.179903 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:30:12.190006 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:12.182032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56226 (local address 127.0.98.62:36877)
0929 16:30:12.182200 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:12.182205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:12.182338 (+ 133us) server_negotiation.cc:407] Connection header received
0929 16:30:12.182443 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:12.182445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:12.182489 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:12.182560 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:12.183259 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.183843 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.184720 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.184944 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.185027 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:12.186524 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:12.186543 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:12.186546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:12.186573 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:12.188176 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:12.188635 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:12.188638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:12.188640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:12.188726 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:12.189057 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:12.189075 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:12.189076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:12.189410 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:12.189521 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:12.189749 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 16:30:12.189853 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:30:12.193133 19588 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:30:12.196393 19588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:30:12.213017 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:12.205794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36089 (local address 127.0.0.1:57256)
0929 16:30:12.206406 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:12.206461 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:12.206591 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:12.207483 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:12.207490 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:12.207516 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:12.207691 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:12.207699 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.209404 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.209409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:12.210982 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:12.210994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.212364 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.212368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:12.212537 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:12.212587 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 16:30:12.212702 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 16:30:12.212850 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":24,"thread_start_us":90,"threads_started":1}
I20260929 16:30:12.214056 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:12.206543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57256 (local address 127.0.98.2:36089)
0929 16:30:12.206948 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:12.206963 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:12.206985 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:30:12.207065 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:12.207070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:12.207276 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:12.207860 (+ 584us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:12.208020 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.209175 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.211176 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.212086 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.213514 (+ 1428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:12.213711 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:12.213775 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 16:30:12.213888 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":178,"threads_started":1}
I20260929 16:30:12.215395 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:12.205964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36541 (local address 127.0.0.1:36162)
0929 16:30:12.207751 (+ 1787us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:12.207766 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:12.207870 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:12.209743 (+ 1873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:12.209746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:12.209765 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:12.209877 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:12.209885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.211100 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.211107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:12.212406 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:12.212414 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.214982 (+ 2568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.214991 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:12.215057 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:12.215073 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 16:30:12.215166 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 16:30:12.215243 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1720,"mutex_wait_us":3,"thread_start_us":132,"threads_started":1}
I20260929 16:30:12.215487 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:12.207130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36162 (local address 127.0.98.3:36541)
0929 16:30:12.207524 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:12.207536 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:12.207977 (+ 441us) server_negotiation.cc:407] Connection header received
0929 16:30:12.208061 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:12.208067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:12.209333 (+ 1266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:12.209557 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:12.210014 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.210767 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.213768 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.214663 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.214743 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:12.214882 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:12.215217 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 16:30:12.215329 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":168,"threads_started":1}
I20260929 16:30:12.216697 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:12.208005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42629 (local address 127.0.0.1:41540)
0929 16:30:12.211357 (+ 3352us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:12.211372 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:12.211462 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:12.212191 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:12.212194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:12.212210 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:12.212310 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:12.212316 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.213687 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.213692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:12.214848 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:12.214858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.216296 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.216300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:12.216401 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:12.216425 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:30:12.216506 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 16:30:12.216575 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3281,"thread_start_us":98,"threads_started":1}
I20260929 16:30:12.216979 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:12.208484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41540 (local address 127.0.98.1:42629)
0929 16:30:12.209018 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:12.209027 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:12.211573 (+ 2546us) server_negotiation.cc:407] Connection header received
0929 16:30:12.211663 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:12.211668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:12.211901 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:12.212043 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:12.212446 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.213558 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.214983 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.216064 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.216189 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:12.216333 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:12.216554 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 16:30:12.216653 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":123,"threads_started":1}
I20260929 16:30:12.217736 19847 tablet_service.cc:1467] Processing CreateTablet for tablet c047a9e08ea945b3b19e6d328754c015 (DEFAULT_TABLE table=test-table [id=aba655761ed842d28583f55ae7195520]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:12.218346 19981 tablet_service.cc:1467] Processing CreateTablet for tablet c047a9e08ea945b3b19e6d328754c015 (DEFAULT_TABLE table=test-table [id=aba655761ed842d28583f55ae7195520]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:12.218921 19847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c047a9e08ea945b3b19e6d328754c015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:12.219317 19981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c047a9e08ea945b3b19e6d328754c015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:12.220119 19713 tablet_service.cc:1467] Processing CreateTablet for tablet c047a9e08ea945b3b19e6d328754c015 (DEFAULT_TABLE table=test-table [id=aba655761ed842d28583f55ae7195520]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:12.221236 19713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c047a9e08ea945b3b19e6d328754c015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:12.225461 20064 tablet_bootstrap.cc:492] T c047a9e08ea945b3b19e6d328754c015 P ad3207ed46f64b50acd38fab19f80535: Bootstrap starting.
I20260929 16:30:12.226940 20065 tablet_bootstrap.cc:492] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341: Bootstrap starting.
I20260929 16:30:12.227957 20064 tablet_bootstrap.cc:654] T c047a9e08ea945b3b19e6d328754c015 P ad3207ed46f64b50acd38fab19f80535: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:12.229298 20065 tablet_bootstrap.cc:654] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:12.229552 20066 tablet_bootstrap.cc:492] T c047a9e08ea945b3b19e6d328754c015 P 7f8a620dfd4c4941b4b0ac88b679861d: Bootstrap starting.
I20260929 16:30:12.229602 20064 log.cc:824] T c047a9e08ea945b3b19e6d328754c015 P ad3207ed46f64b50acd38fab19f80535: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:12.230443 20065 log.cc:824] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:12.232024 20066 tablet_bootstrap.cc:654] T c047a9e08ea945b3b19e6d328754c015 P 7f8a620dfd4c4941b4b0ac88b679861d: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:12.232237 20064 tablet_bootstrap.cc:492] T c047a9e08ea945b3b19e6d328754c015 P ad3207ed46f64b50acd38fab19f80535: No bootstrap required, opened a new log
I20260929 16:30:12.232471 20064 ts_tablet_manager.cc:1397] T c047a9e08ea945b3b19e6d328754c015 P ad3207ed46f64b50acd38fab19f80535: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 16:30:12.232892 20066 log.cc:824] T c047a9e08ea945b3b19e6d328754c015 P 7f8a620dfd4c4941b4b0ac88b679861d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:12.233033 20065 tablet_bootstrap.cc:492] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341: No bootstrap required, opened a new log
I20260929 16:30:12.233289 20065 ts_tablet_manager.cc:1397] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260929 16:30:12.234823 20066 tablet_bootstrap.cc:492] T c047a9e08ea945b3b19e6d328754c015 P 7f8a620dfd4c4941b4b0ac88b679861d: No bootstrap required, opened a new log
I20260929 16:30:12.235026 20066 ts_tablet_manager.cc:1397] T c047a9e08ea945b3b19e6d328754c015 P 7f8a620dfd4c4941b4b0ac88b679861d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:30:12.236704 20064 raft_consensus.cc:348] T c047a9e08ea945b3b19e6d328754c015 P ad3207ed46f64b50acd38fab19f80535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0948f6387e8042e99a883b0bbf352341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "ad3207ed46f64b50acd38fab19f80535" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } peers { permanent_uuid: "7f8a620dfd4c4941b4b0ac88b679861d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42629 } }
I20260929 16:30:12.237042 20064 raft_consensus.cc:374] T c047a9e08ea945b3b19e6d328754c015 P ad3207ed46f64b50acd38fab19f80535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:12.237106 20064 raft_consensus.cc:729] T c047a9e08ea945b3b19e6d328754c015 P ad3207ed46f64b50acd38fab19f80535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad3207ed46f64b50acd38fab19f80535, State: Initialized, Role: FOLLOWER
I20260929 16:30:12.237104 20065 raft_consensus.cc:348] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0948f6387e8042e99a883b0bbf352341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "ad3207ed46f64b50acd38fab19f80535" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } peers { permanent_uuid: "7f8a620dfd4c4941b4b0ac88b679861d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42629 } }
I20260929 16:30:12.237407 20065 raft_consensus.cc:374] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:12.237469 20065 raft_consensus.cc:729] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0948f6387e8042e99a883b0bbf352341, State: Initialized, Role: FOLLOWER
I20260929 16:30:12.237632 20064 consensus_queue.cc:260] T c047a9e08ea945b3b19e6d328754c015 P ad3207ed46f64b50acd38fab19f80535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0948f6387e8042e99a883b0bbf352341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "ad3207ed46f64b50acd38fab19f80535" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } peers { permanent_uuid: "7f8a620dfd4c4941b4b0ac88b679861d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42629 } }
I20260929 16:30:12.237905 20065 consensus_queue.cc:260] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0948f6387e8042e99a883b0bbf352341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "ad3207ed46f64b50acd38fab19f80535" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } peers { permanent_uuid: "7f8a620dfd4c4941b4b0ac88b679861d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42629 } }
I20260929 16:30:12.238435 20047 heartbeater.cc:503] Master 127.0.98.62:36877 was elected leader, sending a full tablet report...
I20260929 16:30:12.238744 20065 ts_tablet_manager.cc:1428] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260929 16:30:12.238744 20064 ts_tablet_manager.cc:1428] T c047a9e08ea945b3b19e6d328754c015 P ad3207ed46f64b50acd38fab19f80535: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:30:12.239133 19913 heartbeater.cc:503] Master 127.0.98.62:36877 was elected leader, sending a full tablet report...
I20260929 16:30:12.239449 20066 raft_consensus.cc:348] T c047a9e08ea945b3b19e6d328754c015 P 7f8a620dfd4c4941b4b0ac88b679861d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0948f6387e8042e99a883b0bbf352341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "ad3207ed46f64b50acd38fab19f80535" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } peers { permanent_uuid: "7f8a620dfd4c4941b4b0ac88b679861d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42629 } }
I20260929 16:30:12.239727 20066 raft_consensus.cc:374] T c047a9e08ea945b3b19e6d328754c015 P 7f8a620dfd4c4941b4b0ac88b679861d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:12.239786 20066 raft_consensus.cc:729] T c047a9e08ea945b3b19e6d328754c015 P 7f8a620dfd4c4941b4b0ac88b679861d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8a620dfd4c4941b4b0ac88b679861d, State: Initialized, Role: FOLLOWER
I20260929 16:30:12.240293 20066 consensus_queue.cc:260] T c047a9e08ea945b3b19e6d328754c015 P 7f8a620dfd4c4941b4b0ac88b679861d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0948f6387e8042e99a883b0bbf352341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "ad3207ed46f64b50acd38fab19f80535" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } peers { permanent_uuid: "7f8a620dfd4c4941b4b0ac88b679861d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42629 } }
I20260929 16:30:12.240963 19779 heartbeater.cc:503] Master 127.0.98.62:36877 was elected leader, sending a full tablet report...
I20260929 16:30:12.241348 20066 ts_tablet_manager.cc:1428] T c047a9e08ea945b3b19e6d328754c015 P 7f8a620dfd4c4941b4b0ac88b679861d: Time spent starting tablet: real 0.006s user 0.002s sys 0.006s
W20260929 16:30:12.405012 20048 tablet.cc:2369] T c047a9e08ea945b3b19e6d328754c015 P ad3207ed46f64b50acd38fab19f80535: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:30:12.409521 19914 tablet.cc:2369] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:30:12.425561 19780 tablet.cc:2369] T c047a9e08ea945b3b19e6d328754c015 P 7f8a620dfd4c4941b4b0ac88b679861d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:30:12.431954 20071 raft_consensus.cc:482] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:30:12.432233 20071 raft_consensus.cc:504] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0948f6387e8042e99a883b0bbf352341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "ad3207ed46f64b50acd38fab19f80535" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } peers { permanent_uuid: "7f8a620dfd4c4941b4b0ac88b679861d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42629 } }
I20260929 16:30:12.433678 20071 leader_election.cc:288] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad3207ed46f64b50acd38fab19f80535 (127.0.98.3:36541), 7f8a620dfd4c4941b4b0ac88b679861d (127.0.98.1:42629)
I20260929 16:30:12.438238 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:12.433845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36541 (local address 127.0.98.2:51187)
0929 16:30:12.434231 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:12.434246 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:12.434390 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:12.434696 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:12.434699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:12.434719 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:12.434999 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:12.435018 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.435967 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.435971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:12.436978 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:12.436989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.437767 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.437771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:12.437876 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:12.437915 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 16:30:12.438000 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 16:30:12.438079 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":88,"threads_started":1}
I20260929 16:30:12.438290 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:12.434007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51187 (local address 127.0.98.3:36541)
0929 16:30:12.434149 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:12.434154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:12.434355 (+ 201us) server_negotiation.cc:407] Connection header received
0929 16:30:12.434523 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:12.434526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:12.434586 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:12.434677 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:12.435236 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.435836 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.437134 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.437637 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.437724 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:12.437779 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:12.438038 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 16:30:12.438134 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 16:30:12.438526 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:12.434206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59937 (local address 127.0.98.1:42629)
0929 16:30:12.434381 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:12.434386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:12.434508 (+ 122us) server_negotiation.cc:407] Connection header received
0929 16:30:12.434720 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:12.434723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:12.434773 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:12.434999 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:12.435442 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.436135 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.437341 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.437867 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.438003 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:12.438063 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:12.438330 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 16:30:12.438407 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:30:12.438768 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:12.434003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42629 (local address 127.0.98.2:59937)
0929 16:30:12.434371 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:12.434387 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:12.434530 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:12.435008 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:12.435011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:12.435038 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:12.435300 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:12.435307 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.436281 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.436283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:12.437127 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:12.437137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.438072 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.438076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:12.438180 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:12.438204 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:30:12.438295 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 16:30:12.438533 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":87,"threads_started":1}
I20260929 16:30:12.439224 20001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c047a9e08ea945b3b19e6d328754c015" candidate_uuid: "0948f6387e8042e99a883b0bbf352341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad3207ed46f64b50acd38fab19f80535" is_pre_election: true
I20260929 16:30:12.439327 19733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c047a9e08ea945b3b19e6d328754c015" candidate_uuid: "0948f6387e8042e99a883b0bbf352341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8a620dfd4c4941b4b0ac88b679861d" is_pre_election: true
I20260929 16:30:12.439595 20001 raft_consensus.cc:2454] T c047a9e08ea945b3b19e6d328754c015 P ad3207ed46f64b50acd38fab19f80535 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0948f6387e8042e99a883b0bbf352341 in term 0.
I20260929 16:30:12.439667 19733 raft_consensus.cc:2454] T c047a9e08ea945b3b19e6d328754c015 P 7f8a620dfd4c4941b4b0ac88b679861d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0948f6387e8042e99a883b0bbf352341 in term 0.
I20260929 16:30:12.440179 19800 leader_election.cc:302] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0948f6387e8042e99a883b0bbf352341, 7f8a620dfd4c4941b4b0ac88b679861d, ad3207ed46f64b50acd38fab19f80535; no voters:
I20260929 16:30:12.440510 20071 raft_consensus.cc:2781] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:30:12.440622 20071 raft_consensus.cc:482] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:30:12.440687 20071 raft_consensus.cc:3037] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:12.441851 20071 raft_consensus.cc:504] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0948f6387e8042e99a883b0bbf352341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "ad3207ed46f64b50acd38fab19f80535" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } peers { permanent_uuid: "7f8a620dfd4c4941b4b0ac88b679861d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42629 } }
I20260929 16:30:12.442214 20071 leader_election.cc:288] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341 [CANDIDATE]: Term 1 election: Requested vote from peers ad3207ed46f64b50acd38fab19f80535 (127.0.98.3:36541), 7f8a620dfd4c4941b4b0ac88b679861d (127.0.98.1:42629)
I20260929 16:30:12.442560 20001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c047a9e08ea945b3b19e6d328754c015" candidate_uuid: "0948f6387e8042e99a883b0bbf352341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad3207ed46f64b50acd38fab19f80535"
I20260929 16:30:12.442704 20001 raft_consensus.cc:3037] T c047a9e08ea945b3b19e6d328754c015 P ad3207ed46f64b50acd38fab19f80535 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:12.442720 19733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c047a9e08ea945b3b19e6d328754c015" candidate_uuid: "0948f6387e8042e99a883b0bbf352341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f8a620dfd4c4941b4b0ac88b679861d"
I20260929 16:30:12.442864 19733 raft_consensus.cc:3037] T c047a9e08ea945b3b19e6d328754c015 P 7f8a620dfd4c4941b4b0ac88b679861d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:12.443903 20001 raft_consensus.cc:2454] T c047a9e08ea945b3b19e6d328754c015 P ad3207ed46f64b50acd38fab19f80535 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0948f6387e8042e99a883b0bbf352341 in term 1.
I20260929 16:30:12.443914 19733 raft_consensus.cc:2454] T c047a9e08ea945b3b19e6d328754c015 P 7f8a620dfd4c4941b4b0ac88b679861d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0948f6387e8042e99a883b0bbf352341 in term 1.
I20260929 16:30:12.444242 19800 leader_election.cc:302] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0948f6387e8042e99a883b0bbf352341, 7f8a620dfd4c4941b4b0ac88b679861d; no voters:
I20260929 16:30:12.444447 20071 raft_consensus.cc:2781] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:12.444720 20071 raft_consensus.cc:686] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341 [term 1 LEADER]: Becoming Leader. State: Replica: 0948f6387e8042e99a883b0bbf352341, State: Running, Role: LEADER
I20260929 16:30:12.445113 20071 consensus_queue.cc:237] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0948f6387e8042e99a883b0bbf352341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } } peers { permanent_uuid: "ad3207ed46f64b50acd38fab19f80535" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } } peers { permanent_uuid: "7f8a620dfd4c4941b4b0ac88b679861d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42629 } }
I20260929 16:30:12.448340 19588 catalog_manager.cc:5495] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0948f6387e8042e99a883b0bbf352341 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0948f6387e8042e99a883b0bbf352341" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0948f6387e8042e99a883b0bbf352341" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad3207ed46f64b50acd38fab19f80535" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8a620dfd4c4941b4b0ac88b679861d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42629 } health_report { overall_health: UNKNOWN } } }
I20260929 16:30:12.506717 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:12.502991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57260 (local address 127.0.98.2:36089)
0929 16:30:12.503150 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:12.503159 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:12.503171 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:30:12.503244 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:12.503246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:12.503297 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:12.503420 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:30:12.503689 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.504243 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.505122 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.505424 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.505542 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:12.505865 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:30:12.506306 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:30:12.506451 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:12.506492 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 16:30:12.506551 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:30:12.513394 20001 raft_consensus.cc:1261] T c047a9e08ea945b3b19e6d328754c015 P ad3207ed46f64b50acd38fab19f80535 [term 1 FOLLOWER]: Refusing update from remote peer 0948f6387e8042e99a883b0bbf352341: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:12.513411 19733 raft_consensus.cc:1261] T c047a9e08ea945b3b19e6d328754c015 P 7f8a620dfd4c4941b4b0ac88b679861d [term 1 FOLLOWER]: Refusing update from remote peer 0948f6387e8042e99a883b0bbf352341: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:12.514159 20071 consensus_queue.cc:1035] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f8a620dfd4c4941b4b0ac88b679861d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:12.514436 20075 consensus_queue.cc:1035] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad3207ed46f64b50acd38fab19f80535" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:12.521620 20078 mvcc.cc:205] Tried to move back new op lower bound from 7334704793646366720 to 7334704793377996800. Current Snapshot: MvccSnapshot[applied={T|T < 7334704793646366720}]
I20260929 16:30:12.524370 20080 mvcc.cc:205] Tried to move back new op lower bound from 7334704793646366720 to 7334704793377996800. Current Snapshot: MvccSnapshot[applied={T|T < 7334704793646366720}]
I20260929 16:30:12.530364 19587 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:30:12.530568 19587 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:30:12.533669 19587 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8c0558821164f4186ed492026f8143f: Sending DeleteTablet for 3 replicas of tablet c047a9e08ea945b3b19e6d328754c015
I20260929 16:30:12.534679 19713 tablet_service.cc:1514] Processing DeleteTablet for tablet c047a9e08ea945b3b19e6d328754c015 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41540
I20260929 16:30:12.534781 19981 tablet_service.cc:1514] Processing DeleteTablet for tablet c047a9e08ea945b3b19e6d328754c015 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36162
I20260929 16:30:12.535421 20087 tablet_replica.cc:330] T c047a9e08ea945b3b19e6d328754c015 P 7f8a620dfd4c4941b4b0ac88b679861d: stopping tablet replica
I20260929 16:30:12.535786 20087 raft_consensus.cc:2229] T c047a9e08ea945b3b19e6d328754c015 P 7f8a620dfd4c4941b4b0ac88b679861d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:30:12.535970 19847 tablet_service.cc:1514] Processing DeleteTablet for tablet c047a9e08ea945b3b19e6d328754c015 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57256
I20260929 16:30:12.536161 20087 raft_consensus.cc:2258] T c047a9e08ea945b3b19e6d328754c015 P 7f8a620dfd4c4941b4b0ac88b679861d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:12.536612 20089 tablet_replica.cc:330] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341: stopping tablet replica
I20260929 16:30:12.536932 20089 raft_consensus.cc:2229] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:30:12.537099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 19650
I20260929 16:30:12.538126 20089 raft_consensus.cc:2258] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:12.538643 20088 tablet_replica.cc:330] T c047a9e08ea945b3b19e6d328754c015 P ad3207ed46f64b50acd38fab19f80535: stopping tablet replica
I20260929 16:30:12.538973 20088 raft_consensus.cc:2229] T c047a9e08ea945b3b19e6d328754c015 P ad3207ed46f64b50acd38fab19f80535 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:30:12.539309 20088 raft_consensus.cc:2258] T c047a9e08ea945b3b19e6d328754c015 P ad3207ed46f64b50acd38fab19f80535 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:12.540627 20088 ts_tablet_manager.cc:1905] T c047a9e08ea945b3b19e6d328754c015 P ad3207ed46f64b50acd38fab19f80535: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:30:12.540627 20089 ts_tablet_manager.cc:1905] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:30:12.543473 20089 ts_tablet_manager.cc:1918] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:30:12.543473 20088 ts_tablet_manager.cc:1918] T c047a9e08ea945b3b19e6d328754c015 P ad3207ed46f64b50acd38fab19f80535: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:30:12.543591 20089 log.cc:1196] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-1/wal/wals/c047a9e08ea945b3b19e6d328754c015
I20260929 16:30:12.543738 20088 log.cc:1196] T c047a9e08ea945b3b19e6d328754c015 P ad3207ed46f64b50acd38fab19f80535: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/c047a9e08ea945b3b19e6d328754c015
I20260929 16:30:12.543927 20089 ts_tablet_manager.cc:1939] T c047a9e08ea945b3b19e6d328754c015 P 0948f6387e8042e99a883b0bbf352341: Deleting consensus metadata
I20260929 16:30:12.544191 20088 ts_tablet_manager.cc:1939] T c047a9e08ea945b3b19e6d328754c015 P ad3207ed46f64b50acd38fab19f80535: Deleting consensus metadata
I20260929 16:30:12.545312 19575 catalog_manager.cc:4841] TS 0948f6387e8042e99a883b0bbf352341 (127.0.98.2:36089): tablet c047a9e08ea945b3b19e6d328754c015 (table test-table [id=aba655761ed842d28583f55ae7195520]) successfully deleted
I20260929 16:30:12.545333 19573 catalog_manager.cc:4841] TS ad3207ed46f64b50acd38fab19f80535 (127.0.98.3:36541): tablet c047a9e08ea945b3b19e6d328754c015 (table test-table [id=aba655761ed842d28583f55ae7195520]) successfully deleted
W20260929 16:30:12.545683 19572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42629 (error 108)
I20260929 16:30:12.546514 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 19784
I20260929 16:30:12.546711 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:12.546425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42629 (local address 127.0.0.1:41552)
0929 16:30:12.546551 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:12.546622 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:30:12.547887 19572 catalog_manager.cc:4584] TS 7f8a620dfd4c4941b4b0ac88b679861d (127.0.98.1:42629): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c047a9e08ea945b3b19e6d328754c015: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42629: connect: Connection refused (error 111)
I20260929 16:30:12.552546 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 19918
I20260929 16:30:12.558702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 19556
2026-09-29T16:30:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2610 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T16:30:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:12Z Disabled control of system clock
I20260929 16:30:12.590193 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32843
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:30:12.706074 20094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:12.706454 20094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:12.706498 20094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:12.711802 20094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:12.711886 20094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:12.711905 20094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:12.711925 20094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:12.711943 20094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:12.717777 20094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32843
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:12.719009 20094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:12.720525 20094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:12.729875 20099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:12.729925 20102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:12.729805 20100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:12.730116 20094 server_base.cc:1034] running on GCE node
I20260929 16:30:12.730921 20094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:12.732120 20094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:12.733325 20094 hybrid_clock.cc:648] HybridClock initialized: now 1790699412733302 us; error 31 us; skew 500 ppm
I20260929 16:30:12.735809 20094 webserver.cc:460] Webserver started at http://127.0.98.62:37925/ using document root <none> and password file <none>
I20260929 16:30:12.736424 20094 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:12.736500 20094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:12.736703 20094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:12.739230 20094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "e7c40fceec864a978618b95f01ff3598"
format_stamp: "Formatted at 2026-09-29 16:30:12 on dist-test-slave-vq9v"
server_key: "995bfd08871b9ca9ccb74dee8220e28c"
server_key_iv: "54cd75c78c97d007045ce2542b4aa16e"
server_key_version: "encryptionkey@0"
I20260929 16:30:12.739864 20094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "e7c40fceec864a978618b95f01ff3598"
format_stamp: "Formatted at 2026-09-29 16:30:12 on dist-test-slave-vq9v"
server_key: "995bfd08871b9ca9ccb74dee8220e28c"
server_key_iv: "54cd75c78c97d007045ce2542b4aa16e"
server_key_version: "encryptionkey@0"
I20260929 16:30:12.744264 20094 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 16:30:12.747351 20108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:12.748910 20094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:30:12.749054 20094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "e7c40fceec864a978618b95f01ff3598"
format_stamp: "Formatted at 2026-09-29 16:30:12 on dist-test-slave-vq9v"
server_key: "995bfd08871b9ca9ccb74dee8220e28c"
server_key_iv: "54cd75c78c97d007045ce2542b4aa16e"
server_key_version: "encryptionkey@0"
I20260929 16:30:12.749178 20094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:12.761270 20094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:12.762086 20094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:12.762283 20094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:12.770875 20094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32843
I20260929 16:30:12.770891 20160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32843 every 8 connection(s)
I20260929 16:30:12.772007 20094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:12.775252 20161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:12.776522 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 20094
I20260929 16:30:12.776648 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:12.776978 392 external_mini_cluster.cc:1428] Setting key b371d722ad31b683e69d67c4a80ac8a6
I20260929 16:30:12.781719 20161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598: Bootstrap starting.
I20260929 16:30:12.783967 20161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:12.784777 20161 log.cc:824] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:12.786960 20161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598: No bootstrap required, opened a new log
I20260929 16:30:12.786984 20164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:12.778538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46464 (local address 127.0.98.62:32843)
0929 16:30:12.779210 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:12.779225 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:12.779260 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:30:12.779973 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:12.779991 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:12.780410 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:12.780791 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:12.781312 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.782372 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.783283 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:12.783773 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:12.783961 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:12.784855 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:12.784859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:12.784873 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:12.785425 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:12.785498 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:12.785530 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:12.785646 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:12.785848 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 16:30:12.786149 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":145,"threads_started":1}
I20260929 16:30:12.790982 20161 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c40fceec864a978618b95f01ff3598" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32843 } }
I20260929 16:30:12.791308 20161 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:12.791395 20161 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c40fceec864a978618b95f01ff3598, State: Initialized, Role: FOLLOWER
I20260929 16:30:12.791848 20161 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c40fceec864a978618b95f01ff3598" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32843 } }
I20260929 16:30:12.791999 20161 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:12.792068 20161 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:12.792169 20161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:12.793150 20161 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c40fceec864a978618b95f01ff3598" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32843 } }
I20260929 16:30:12.793532 20161 leader_election.cc:302] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7c40fceec864a978618b95f01ff3598; no voters:
I20260929 16:30:12.793923 20161 leader_election.cc:288] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:12.794054 20166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:12.794322 20166 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598 [term 1 LEADER]: Becoming Leader. State: Replica: e7c40fceec864a978618b95f01ff3598, State: Running, Role: LEADER
I20260929 16:30:12.794636 20166 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c40fceec864a978618b95f01ff3598" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32843 } }
I20260929 16:30:12.795256 20161 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:12.796301 20167 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7c40fceec864a978618b95f01ff3598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c40fceec864a978618b95f01ff3598" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32843 } } }
I20260929 16:30:12.796445 20167 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:12.796280 20168 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7c40fceec864a978618b95f01ff3598. Latest consensus state: current_term: 1 leader_uuid: "e7c40fceec864a978618b95f01ff3598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c40fceec864a978618b95f01ff3598" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32843 } } }
I20260929 16:30:12.796677 20168 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:12.796968 20175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:12.799985 20175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:12.806391 20175 catalog_manager.cc:1347] Generated new cluster ID: 784930622bee4d39874f13cdfbfd0cef
I20260929 16:30:12.806485 20175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:12.831854 20175 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:12.832849 20175 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:12.837953 20175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598: Generated new TSK 0
I20260929 16:30:12.838673 20175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:12.851562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32843
--builtin_ntp_servers=127.0.98.20:36167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:30:12.976924 20185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:12.977319 20185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:12.977381 20185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:12.981734 20185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:12.981804 20185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:12.981871 20185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 16:30:12.986843 20185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:12.987938 20185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:12.989388 20185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:12.999320 20190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:12.999415 20185 server_base.cc:1034] running on GCE node
W20260929 16:30:12.999451 20191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:12.999320 20193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:13.000102 20185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:13.000797 20185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:13.002009 20185 hybrid_clock.cc:648] HybridClock initialized: now 1790699413001960 us; error 64 us; skew 500 ppm
I20260929 16:30:13.003988 20185 webserver.cc:460] Webserver started at http://127.0.98.1:46337/ using document root <none> and password file <none>
I20260929 16:30:13.004607 20185 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:13.004709 20185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:13.004971 20185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:13.007220 20185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-0/data/instance:
uuid: "af2f174ce42d4e838b5cf89bb466b0d5"
format_stamp: "Formatted at 2026-09-29 16:30:13 on dist-test-slave-vq9v"
server_key: "f16e70779d0df70052eb0dc80d16c7eb"
server_key_iv: "ef9d2292c9ab7917cd09defd3b5479af"
server_key_version: "encryptionkey@0"
I20260929 16:30:13.007831 20185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-0/wal/instance:
uuid: "af2f174ce42d4e838b5cf89bb466b0d5"
format_stamp: "Formatted at 2026-09-29 16:30:13 on dist-test-slave-vq9v"
server_key: "f16e70779d0df70052eb0dc80d16c7eb"
server_key_iv: "ef9d2292c9ab7917cd09defd3b5479af"
server_key_version: "encryptionkey@0"
I20260929 16:30:13.011796 20185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 16:30:13.014580 20199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:13.015908 20185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:30:13.016077 20185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-0/wal
uuid: "af2f174ce42d4e838b5cf89bb466b0d5"
format_stamp: "Formatted at 2026-09-29 16:30:13 on dist-test-slave-vq9v"
server_key: "f16e70779d0df70052eb0dc80d16c7eb"
server_key_iv: "ef9d2292c9ab7917cd09defd3b5479af"
server_key_version: "encryptionkey@0"
I20260929 16:30:13.016211 20185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:13.029748 20185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:13.030473 20185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:13.030685 20185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:13.031302 20185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:13.032296 20185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:13.032368 20185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:13.032442 20185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:13.032488 20185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:13.042379 20185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34929
I20260929 16:30:13.042419 20312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34929 every 8 connection(s)
I20260929 16:30:13.043416 20185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-0/data/info.pb
I20260929 16:30:13.048969 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 20185
I20260929 16:30:13.049085 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-0/wal/instance
I20260929 16:30:13.049467 392 external_mini_cluster.cc:1428] Setting key db445a5db727dd2a78c127e2273cedc1
I20260929 16:30:13.051676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32843
--builtin_ntp_servers=127.0.98.20:36167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:30:13.053645 20164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.045645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58849 (local address 127.0.98.62:32843)
0929 16:30:13.045818 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:13.045822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:13.046958 (+ 1136us) server_negotiation.cc:407] Connection header received
0929 16:30:13.048402 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:13.048406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:13.048465 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:13.048551 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:13.049664 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.050223 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.051448 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.051679 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.051804 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.053030 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:13.053034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:13.053037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:13.053050 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:13.053079 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:13.053087 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:13.053165 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:13.053427 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 16:30:13.053501 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:30:13.054641 20315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.046112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32843 (local address 127.0.98.1:58849)
0929 16:30:13.046801 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:13.046844 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:13.048161 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:13.048742 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:13.048749 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:13.048837 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:13.049456 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:13.049471 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.050360 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.050364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:13.051279 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:13.051288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.051846 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.051851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:13.052077 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.052756 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:30:13.052757 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:13.052801 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:30:13.052870 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:13.053178 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:13.053190 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:13.053250 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 16:30:13.053428 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 16:30:13.053739 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":127,"threads_started":1}
I20260929 16:30:13.056035 20313 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32843
I20260929 16:30:13.056294 20313 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:13.056854 20313 heartbeater.cc:511] Master 127.0.98.62:32843 requested a full tablet report, sending...
I20260929 16:30:13.058615 20125 ts_manager.cc:194] Registered new tserver with Master: af2f174ce42d4e838b5cf89bb466b0d5 (127.0.98.1:34929)
I20260929 16:30:13.059831 20125 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58849
W20260929 16:30:13.173918 20316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:13.174319 20316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:13.174407 20316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:13.178905 20316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:13.179009 20316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:13.179111 20316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 16:30:13.185042 20316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:13.186292 20316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:13.187776 20316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:13.198163 20321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:13.198156 20322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:13.198207 20324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:13.198510 20316 server_base.cc:1034] running on GCE node
I20260929 16:30:13.198963 20316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:13.199589 20316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:13.200763 20316 hybrid_clock.cc:648] HybridClock initialized: now 1790699413200745 us; error 33 us; skew 500 ppm
I20260929 16:30:13.202827 20316 webserver.cc:460] Webserver started at http://127.0.98.2:39437/ using document root <none> and password file <none>
I20260929 16:30:13.203496 20316 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:13.203574 20316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:13.203751 20316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:13.206089 20316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-1/data/instance:
uuid: "6e9e0e2eb54a4149bd265c6cc164f974"
format_stamp: "Formatted at 2026-09-29 16:30:13 on dist-test-slave-vq9v"
server_key: "6c03c8542b6957fbee842018eb87db7c"
server_key_iv: "0b986fdfa0a888488b0792e4bf92b0f0"
server_key_version: "encryptionkey@0"
I20260929 16:30:13.206650 20316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6e9e0e2eb54a4149bd265c6cc164f974"
format_stamp: "Formatted at 2026-09-29 16:30:13 on dist-test-slave-vq9v"
server_key: "6c03c8542b6957fbee842018eb87db7c"
server_key_iv: "0b986fdfa0a888488b0792e4bf92b0f0"
server_key_version: "encryptionkey@0"
I20260929 16:30:13.210862 20316 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:30:13.213939 20330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:13.215526 20316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:30:13.215714 20316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-1/wal
uuid: "6e9e0e2eb54a4149bd265c6cc164f974"
format_stamp: "Formatted at 2026-09-29 16:30:13 on dist-test-slave-vq9v"
server_key: "6c03c8542b6957fbee842018eb87db7c"
server_key_iv: "0b986fdfa0a888488b0792e4bf92b0f0"
server_key_version: "encryptionkey@0"
I20260929 16:30:13.215867 20316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:13.237296 20316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:13.238091 20316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:13.238322 20316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:13.238988 20316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:13.240031 20316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:13.240082 20316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:13.240125 20316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:13.240185 20316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:13.249449 20316 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42391
I20260929 16:30:13.249488 20443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42391 every 8 connection(s)
I20260929 16:30:13.250449 20316 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-1/data/info.pb
I20260929 16:30:13.259536 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 20316
I20260929 16:30:13.259654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-1/wal/instance
I20260929 16:30:13.259948 392 external_mini_cluster.cc:1428] Setting key 4629e27e01437dd1c4ae0a32c1adf156
I20260929 16:30:13.261180 20164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.252445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34265 (local address 127.0.98.62:32843)
0929 16:30:13.252595 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:13.252599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:13.254016 (+ 1417us) server_negotiation.cc:407] Connection header received
0929 16:30:13.255670 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:13.255674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:13.255737 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:13.255859 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:13.256859 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.257523 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.258806 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.259051 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.259200 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.260575 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:13.260583 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:13.260590 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:13.260609 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:13.260635 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:13.260649 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:13.260729 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:13.260944 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 16:30:13.261019 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:30:13.262243 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.252947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32843 (local address 127.0.98.2:34265)
0929 16:30:13.253853 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:13.253899 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:13.255429 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:13.256053 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:13.256061 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:13.256163 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:13.256707 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:13.256720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.257674 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.257678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:13.258654 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:13.258674 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.259223 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.259228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:13.259429 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.260230 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:30:13.260232 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:13.260289 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:30:13.260380 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:13.260756 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:13.260766 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:13.260772 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:30:13.260916 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 16:30:13.261176 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":120,"threads_started":1}
I20260929 16:30:13.262511 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32843
--builtin_ntp_servers=127.0.98.20:36167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:30:13.263440 20444 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32843
I20260929 16:30:13.263731 20444 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:13.264369 20444 heartbeater.cc:511] Master 127.0.98.62:32843 requested a full tablet report, sending...
I20260929 16:30:13.265497 20125 ts_manager.cc:194] Registered new tserver with Master: 6e9e0e2eb54a4149bd265c6cc164f974 (127.0.98.2:42391)
I20260929 16:30:13.266085 20125 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:34265
W20260929 16:30:13.386440 20447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:13.386813 20447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:13.386897 20447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:13.391695 20447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:13.391820 20447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:13.391929 20447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 16:30:13.397768 20447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:13.399142 20447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:13.400679 20447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:13.411877 20455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:13.411943 20447 server_base.cc:1034] running on GCE node
W20260929 16:30:13.411866 20452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:13.411866 20453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:13.412617 20447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:13.413486 20447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:13.414675 20447 hybrid_clock.cc:648] HybridClock initialized: now 1790699413414638 us; error 53 us; skew 500 ppm
I20260929 16:30:13.416900 20447 webserver.cc:460] Webserver started at http://127.0.98.3:34543/ using document root <none> and password file <none>
I20260929 16:30:13.417625 20447 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:13.417696 20447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:13.417979 20447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:13.420342 20447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-2/data/instance:
uuid: "26d44557b7464a37bf2870582072aa2d"
format_stamp: "Formatted at 2026-09-29 16:30:13 on dist-test-slave-vq9v"
server_key: "51684ef5b17442bb388373ef04b96919"
server_key_iv: "7ff97139fbc3201ebfefe009d742dcce"
server_key_version: "encryptionkey@0"
I20260929 16:30:13.420969 20447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-2/wal/instance:
uuid: "26d44557b7464a37bf2870582072aa2d"
format_stamp: "Formatted at 2026-09-29 16:30:13 on dist-test-slave-vq9v"
server_key: "51684ef5b17442bb388373ef04b96919"
server_key_iv: "7ff97139fbc3201ebfefe009d742dcce"
server_key_version: "encryptionkey@0"
I20260929 16:30:13.425079 20447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.002s
I20260929 16:30:13.427996 20461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:13.429541 20447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:13.429737 20447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-2/wal
uuid: "26d44557b7464a37bf2870582072aa2d"
format_stamp: "Formatted at 2026-09-29 16:30:13 on dist-test-slave-vq9v"
server_key: "51684ef5b17442bb388373ef04b96919"
server_key_iv: "7ff97139fbc3201ebfefe009d742dcce"
server_key_version: "encryptionkey@0"
I20260929 16:30:13.429875 20447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:13.451108 20447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:13.451920 20447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:13.452158 20447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:13.452855 20447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:30:13.454116 20447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:30:13.454175 20447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:13.454248 20447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:30:13.454305 20447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:13.464334 20447 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39097
I20260929 16:30:13.464437 20574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39097 every 8 connection(s)
I20260929 16:30:13.465547 20447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-2/data/info.pb
I20260929 16:30:13.470427 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 20447
I20260929 16:30:13.470558 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-2/wal/instance
I20260929 16:30:13.470909 392 external_mini_cluster.cc:1428] Setting key 7b4264df9b5e689112a959c52e934333
I20260929 16:30:13.476291 20164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.467706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56047 (local address 127.0.98.62:32843)
0929 16:30:13.467894 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:13.467899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:13.469050 (+ 1151us) server_negotiation.cc:407] Connection header received
0929 16:30:13.470776 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:13.470779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:13.470835 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:13.470917 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:13.472208 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.472983 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.474242 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.474489 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.474604 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.475776 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:13.475780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:13.475783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:13.475796 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:13.475817 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:13.475833 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:13.475915 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:13.476085 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 16:30:13.476148 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:30:13.477289 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.468110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32843 (local address 127.0.98.3:56047)
0929 16:30:13.468888 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:13.468931 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:13.470426 (+ 1495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:13.471165 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:13.471176 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:13.471296 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:13.472045 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:13.472066 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.473112 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.473122 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:13.474109 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:13.474124 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.474649 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.474653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:13.474835 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.475509 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:30:13.475511 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:13.475553 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:30:13.475621 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:13.475916 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:13.475927 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:13.475936 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:30:13.476075 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 16:30:13.476387 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":142,"threads_started":1}
I20260929 16:30:13.478451 20575 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32843
I20260929 16:30:13.478719 20575 heartbeater.cc:464] Registering TS with master...
I20260929 16:30:13.479274 20575 heartbeater.cc:511] Master 127.0.98.62:32843 requested a full tablet report, sending...
I20260929 16:30:13.480430 20125 ts_manager.cc:194] Registered new tserver with Master: 26d44557b7464a37bf2870582072aa2d (127.0.98.3:39097)
I20260929 16:30:13.481015 20125 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:56047
I20260929 16:30:13.484830 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:30:13.490867 20164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.486947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46472 (local address 127.0.98.62:32843)
0929 16:30:13.487094 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:13.487099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:13.487291 (+ 192us) server_negotiation.cc:407] Connection header received
0929 16:30:13.487445 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:13.487449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:13.487496 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:13.487582 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:13.487873 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.488386 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.489262 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.489481 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.489593 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.490426 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:13.490430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:13.490432 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:13.490452 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:13.490474 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:13.490482 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:13.490568 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:13.490655 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 16:30:13.490718 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:30:13.493903 20125 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:30:13.497432 20125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:30:13.512586 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.506674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34929 (local address 127.0.0.1:45582)
0929 16:30:13.507171 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:13.507222 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:13.507350 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:13.508280 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:13.508286 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:13.508312 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:13.508466 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:13.508474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.509946 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.509953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:13.511270 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:13.511281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.512158 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.512166 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:13.512255 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.512286 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:30:13.512366 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 16:30:13.512461 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":104,"threads_started":1}
I20260929 16:30:13.512734 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.506866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45582 (local address 127.0.98.1:34929)
0929 16:30:13.507566 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:13.507581 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:13.507611 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:30:13.507700 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:13.507705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:13.507930 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:13.508265 (+ 335us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:13.508614 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.509822 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.511412 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.511997 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.512162 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.512388 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:13.512446 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 16:30:13.512550 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":178,"threads_started":1}
I20260929 16:30:13.514056 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.506885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42391 (local address 127.0.0.1:42952)
0929 16:30:13.507714 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:13.507728 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:13.507809 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:13.509348 (+ 1539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:13.509352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:13.509372 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:13.509513 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:13.509521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.510863 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.510866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:13.512011 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:13.512017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.513405 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.513408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:13.513482 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.513501 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 16:30:13.513617 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 16:30:13.513696 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":21,"thread_start_us":110,"threads_started":1}
I20260929 16:30:13.514412 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.507454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39097 (local address 127.0.0.1:37792)
0929 16:30:13.508490 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:13.508509 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:13.508586 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:13.509139 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:13.509144 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:13.509165 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:13.509302 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:13.509309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.510738 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.510741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:13.511991 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:13.512001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.513999 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.514004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:13.514095 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.514116 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:30:13.514198 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 16:30:13.514268 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"mutex_wait_us":27,"thread_start_us":115,"threads_started":1}
I20260929 16:30:13.514772 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.508043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.0.98.2:42391)
0929 16:30:13.508811 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:13.508819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:13.508838 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:30:13.508897 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:13.508900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:13.509039 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:13.509204 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:13.509663 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.510747 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.512149 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.512992 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.513114 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.513342 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:13.514387 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0929 16:30:13.514643 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":184,"threads_started":1}
I20260929 16:30:13.514869 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.508083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37792 (local address 127.0.98.3:39097)
0929 16:30:13.508625 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:13.508640 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:13.508667 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:30:13.508748 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:13.508754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:13.508967 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:13.509145 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:13.509440 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.510587 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.512825 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.513655 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.513770 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.513932 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:13.514580 (+ 648us) server_negotiation.cc:299] Negotiation successful
0929 16:30:13.514681 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":161,"threads_started":1}
I20260929 16:30:13.516431 20247 tablet_service.cc:1467] Processing CreateTablet for tablet 9139f9c8e0b849849f8586851d343f2b (DEFAULT_TABLE table=test-table [id=70f3ee017a7640c5ba9c17446067fac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:13.517657 20247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9139f9c8e0b849849f8586851d343f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:13.517354 20378 tablet_service.cc:1467] Processing CreateTablet for tablet 9139f9c8e0b849849f8586851d343f2b (DEFAULT_TABLE table=test-table [id=70f3ee017a7640c5ba9c17446067fac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:13.517467 20509 tablet_service.cc:1467] Processing CreateTablet for tablet 9139f9c8e0b849849f8586851d343f2b (DEFAULT_TABLE table=test-table [id=70f3ee017a7640c5ba9c17446067fac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:30:13.518322 20378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9139f9c8e0b849849f8586851d343f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:13.518396 20509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9139f9c8e0b849849f8586851d343f2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:13.525287 20592 tablet_bootstrap.cc:492] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974: Bootstrap starting.
I20260929 16:30:13.526556 20594 tablet_bootstrap.cc:492] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d: Bootstrap starting.
I20260929 16:30:13.527647 20592 tablet_bootstrap.cc:654] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:13.528337 20594 tablet_bootstrap.cc:654] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:13.528422 20592 log.cc:824] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:13.529160 20594 log.cc:824] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:13.529469 20593 tablet_bootstrap.cc:492] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5: Bootstrap starting.
I20260929 16:30:13.531314 20593 tablet_bootstrap.cc:654] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:13.532065 20593 log.cc:824] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:13.532920 20592 tablet_bootstrap.cc:492] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974: No bootstrap required, opened a new log
I20260929 16:30:13.533149 20592 ts_tablet_manager.cc:1397] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 16:30:13.534170 20593 tablet_bootstrap.cc:492] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5: No bootstrap required, opened a new log
I20260929 16:30:13.534171 20594 tablet_bootstrap.cc:492] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d: No bootstrap required, opened a new log
I20260929 16:30:13.534390 20594 ts_tablet_manager.cc:1397] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 16:30:13.534390 20593 ts_tablet_manager.cc:1397] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 16:30:13.538880 20592 raft_consensus.cc:348] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2f174ce42d4e838b5cf89bb466b0d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34929 } } peers { permanent_uuid: "6e9e0e2eb54a4149bd265c6cc164f974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "26d44557b7464a37bf2870582072aa2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39097 } }
I20260929 16:30:13.538875 20593 raft_consensus.cc:348] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2f174ce42d4e838b5cf89bb466b0d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34929 } } peers { permanent_uuid: "6e9e0e2eb54a4149bd265c6cc164f974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "26d44557b7464a37bf2870582072aa2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39097 } }
I20260929 16:30:13.538875 20594 raft_consensus.cc:348] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2f174ce42d4e838b5cf89bb466b0d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34929 } } peers { permanent_uuid: "6e9e0e2eb54a4149bd265c6cc164f974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "26d44557b7464a37bf2870582072aa2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39097 } }
I20260929 16:30:13.539202 20594 raft_consensus.cc:374] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:13.539202 20593 raft_consensus.cc:374] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:13.539203 20592 raft_consensus.cc:374] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:13.539265 20592 raft_consensus.cc:729] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e9e0e2eb54a4149bd265c6cc164f974, State: Initialized, Role: FOLLOWER
I20260929 16:30:13.539265 20593 raft_consensus.cc:729] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af2f174ce42d4e838b5cf89bb466b0d5, State: Initialized, Role: FOLLOWER
I20260929 16:30:13.539265 20594 raft_consensus.cc:729] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d44557b7464a37bf2870582072aa2d, State: Initialized, Role: FOLLOWER
I20260929 16:30:13.539767 20592 consensus_queue.cc:260] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2f174ce42d4e838b5cf89bb466b0d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34929 } } peers { permanent_uuid: "6e9e0e2eb54a4149bd265c6cc164f974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "26d44557b7464a37bf2870582072aa2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39097 } }
I20260929 16:30:13.539760 20593 consensus_queue.cc:260] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2f174ce42d4e838b5cf89bb466b0d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34929 } } peers { permanent_uuid: "6e9e0e2eb54a4149bd265c6cc164f974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "26d44557b7464a37bf2870582072aa2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39097 } }
I20260929 16:30:13.539760 20594 consensus_queue.cc:260] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2f174ce42d4e838b5cf89bb466b0d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34929 } } peers { permanent_uuid: "6e9e0e2eb54a4149bd265c6cc164f974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "26d44557b7464a37bf2870582072aa2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39097 } }
I20260929 16:30:13.540889 20593 ts_tablet_manager.cc:1428] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:30:13.540889 20594 ts_tablet_manager.cc:1428] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 16:30:13.540915 20575 heartbeater.cc:503] Master 127.0.98.62:32843 was elected leader, sending a full tablet report...
I20260929 16:30:13.540915 20313 heartbeater.cc:503] Master 127.0.98.62:32843 was elected leader, sending a full tablet report...
I20260929 16:30:13.541136 20592 ts_tablet_manager.cc:1428] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260929 16:30:13.541249 20444 heartbeater.cc:503] Master 127.0.98.62:32843 was elected leader, sending a full tablet report...
W20260929 16:30:13.544780 20314 tablet.cc:2369] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:30:13.716686 20576 tablet.cc:2369] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:30:13.751689 20445 tablet.cc:2369] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:30:13.979163 20600 raft_consensus.cc:482] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:30:13.979454 20600 raft_consensus.cc:504] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2f174ce42d4e838b5cf89bb466b0d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34929 } } peers { permanent_uuid: "6e9e0e2eb54a4149bd265c6cc164f974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "26d44557b7464a37bf2870582072aa2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39097 } }
I20260929 16:30:13.980664 20600 leader_election.cc:288] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e9e0e2eb54a4149bd265c6cc164f974 (127.0.98.2:42391), 26d44557b7464a37bf2870582072aa2d (127.0.98.3:39097)
I20260929 16:30:13.984112 20598 raft_consensus.cc:482] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:30:13.984302 20598 raft_consensus.cc:504] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2f174ce42d4e838b5cf89bb466b0d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34929 } } peers { permanent_uuid: "6e9e0e2eb54a4149bd265c6cc164f974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "26d44557b7464a37bf2870582072aa2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39097 } }
I20260929 16:30:13.984974 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.981258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37579 (local address 127.0.98.2:42391)
0929 16:30:13.981435 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:13.981441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:13.981459 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:30:13.981524 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:13.981527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:13.981580 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:13.981680 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:13.981986 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.982504 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.983617 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.984249 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.984354 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.984438 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:13.984780 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 16:30:13.984854 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 16:30:13.985002 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.980950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42391 (local address 127.0.98.1:37579)
0929 16:30:13.981253 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:13.981271 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:13.981375 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:13.981705 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:13.981708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:13.981728 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:13.981859 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:13.981864 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.982621 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.982627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:13.983484 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:13.983492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.984500 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.984504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:13.984602 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.984648 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 16:30:13.984751 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 16:30:13.984823 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":20,"thread_start_us":112,"threads_started":1}
I20260929 16:30:13.985113 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.981536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40873 (local address 127.0.98.3:39097)
0929 16:30:13.981707 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:13.981711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:13.981867 (+ 156us) server_negotiation.cc:407] Connection header received
0929 16:30:13.981994 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:13.981997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:13.982040 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:13.982128 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:13.982400 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.982944 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.984012 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.984553 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.984658 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.984716 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:13.984923 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 16:30:13.984996 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 16:30:13.985574 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.981382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39097 (local address 127.0.98.1:40873)
0929 16:30:13.981769 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:13.981780 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:13.981906 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:13.982138 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:13.982141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:13.982156 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:13.982293 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:13.982298 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.983047 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.983054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:13.983909 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:13.983921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.984681 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.984687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:13.984793 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.984812 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 16:30:13.985268 (+ 456us) client_negotiation.cc:240] Negotiation successful
0929 16:30:13.985376 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":20,"thread_start_us":93,"threads_started":1}
I20260929 16:30:13.985831 20398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9139f9c8e0b849849f8586851d343f2b" candidate_uuid: "af2f174ce42d4e838b5cf89bb466b0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e9e0e2eb54a4149bd265c6cc164f974" is_pre_election: true
I20260929 16:30:13.986052 20398 raft_consensus.cc:2454] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af2f174ce42d4e838b5cf89bb466b0d5 in term 0.
I20260929 16:30:13.986136 20529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9139f9c8e0b849849f8586851d343f2b" candidate_uuid: "af2f174ce42d4e838b5cf89bb466b0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d44557b7464a37bf2870582072aa2d" is_pre_election: true
I20260929 16:30:13.986340 20598 leader_election.cc:288] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af2f174ce42d4e838b5cf89bb466b0d5 (127.0.98.1:34929), 26d44557b7464a37bf2870582072aa2d (127.0.98.3:39097)
I20260929 16:30:13.986423 20529 raft_consensus.cc:2454] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af2f174ce42d4e838b5cf89bb466b0d5 in term 0.
I20260929 16:30:13.986966 20201 leader_election.cc:302] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d44557b7464a37bf2870582072aa2d, af2f174ce42d4e838b5cf89bb466b0d5; no voters:
I20260929 16:30:13.987339 20600 raft_consensus.cc:2781] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:30:13.987442 20600 raft_consensus.cc:482] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:30:13.987498 20600 raft_consensus.cc:3037] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:13.988840 20600 raft_consensus.cc:504] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2f174ce42d4e838b5cf89bb466b0d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34929 } } peers { permanent_uuid: "6e9e0e2eb54a4149bd265c6cc164f974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "26d44557b7464a37bf2870582072aa2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39097 } }
I20260929 16:30:13.989182 20600 leader_election.cc:288] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5 [CANDIDATE]: Term 1 election: Requested vote from peers 6e9e0e2eb54a4149bd265c6cc164f974 (127.0.98.2:42391), 26d44557b7464a37bf2870582072aa2d (127.0.98.3:39097)
I20260929 16:30:13.989544 20398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9139f9c8e0b849849f8586851d343f2b" candidate_uuid: "af2f174ce42d4e838b5cf89bb466b0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e9e0e2eb54a4149bd265c6cc164f974"
I20260929 16:30:13.989657 20398 raft_consensus.cc:3037] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:13.989622 20529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9139f9c8e0b849849f8586851d343f2b" candidate_uuid: "af2f174ce42d4e838b5cf89bb466b0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d44557b7464a37bf2870582072aa2d"
I20260929 16:30:13.989765 20529 raft_consensus.cc:3037] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:13.990617 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.986218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36839 (local address 127.0.98.1:34929)
0929 16:30:13.986328 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:13.986334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:13.986633 (+ 299us) server_negotiation.cc:407] Connection header received
0929 16:30:13.986838 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:13.986841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:13.986897 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:13.986966 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:13.987299 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.987923 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.989378 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.989967 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.990041 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.990097 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:13.990421 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 16:30:13.990489 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 16:30:13.990808 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.986281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60889 (local address 127.0.98.3:39097)
0929 16:30:13.986431 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:13.986435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:13.987019 (+ 584us) server_negotiation.cc:407] Connection header received
0929 16:30:13.987220 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:13.987225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:13.987276 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:13.987346 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:30:13.987846 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.988370 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.989575 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.990148 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.990223 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.990276 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:13.990610 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 16:30:13.990676 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:30:13.990993 20398 raft_consensus.cc:2454] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af2f174ce42d4e838b5cf89bb466b0d5 in term 1.
I20260929 16:30:13.991050 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.985924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34929 (local address 127.0.98.2:36839)
0929 16:30:13.986551 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:13.986567 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:13.986707 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:13.986983 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:13.986986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:13.987002 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:13.987134 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:13.987143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.988034 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.988038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:13.989227 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:13.989239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.990192 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.990195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:13.990288 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.990313 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:30:13.990776 (+ 463us) client_negotiation.cc:240] Negotiation successful
0929 16:30:13.990867 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":76,"threads_started":1}
I20260929 16:30:13.991071 20529 raft_consensus.cc:2454] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af2f174ce42d4e838b5cf89bb466b0d5 in term 1.
I20260929 16:30:13.991207 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:13.985973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39097 (local address 127.0.98.2:60889)
0929 16:30:13.986945 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:13.986960 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:13.987095 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:13.987575 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:13.987578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:13.987596 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:30:13.987714 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:13.987723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.988483 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.988488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:13.989450 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:13.989461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:13.990379 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:13.990385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:13.990478 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:13.990500 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:30:13.990980 (+ 480us) client_negotiation.cc:240] Negotiation successful
0929 16:30:13.991069 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"thread_start_us":78,"threads_started":1}
I20260929 16:30:13.991806 20201 leader_election.cc:302] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e9e0e2eb54a4149bd265c6cc164f974, af2f174ce42d4e838b5cf89bb466b0d5; no voters:
I20260929 16:30:13.991731 20267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9139f9c8e0b849849f8586851d343f2b" candidate_uuid: "6e9e0e2eb54a4149bd265c6cc164f974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af2f174ce42d4e838b5cf89bb466b0d5" is_pre_election: true
I20260929 16:30:13.991947 20529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9139f9c8e0b849849f8586851d343f2b" candidate_uuid: "6e9e0e2eb54a4149bd265c6cc164f974" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d44557b7464a37bf2870582072aa2d" is_pre_election: true
I20260929 16:30:13.992062 20267 raft_consensus.cc:2379] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e9e0e2eb54a4149bd265c6cc164f974 in current term 1: Already voted for candidate af2f174ce42d4e838b5cf89bb466b0d5 in this term.
I20260929 16:30:13.992108 20529 raft_consensus.cc:2379] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e9e0e2eb54a4149bd265c6cc164f974 in current term 1: Already voted for candidate af2f174ce42d4e838b5cf89bb466b0d5 in this term.
I20260929 16:30:13.992251 20600 raft_consensus.cc:2781] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:13.992477 20600 raft_consensus.cc:686] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5 [term 1 LEADER]: Becoming Leader. State: Replica: af2f174ce42d4e838b5cf89bb466b0d5, State: Running, Role: LEADER
I20260929 16:30:13.992681 20332 leader_election.cc:302] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e9e0e2eb54a4149bd265c6cc164f974; no voters: 26d44557b7464a37bf2870582072aa2d, af2f174ce42d4e838b5cf89bb466b0d5
I20260929 16:30:13.992805 20600 consensus_queue.cc:237] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2f174ce42d4e838b5cf89bb466b0d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34929 } } peers { permanent_uuid: "6e9e0e2eb54a4149bd265c6cc164f974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "26d44557b7464a37bf2870582072aa2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39097 } }
I20260929 16:30:13.992945 20598 raft_consensus.cc:2726] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 16:30:13.996100 20125 catalog_manager.cc:5495] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to af2f174ce42d4e838b5cf89bb466b0d5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "af2f174ce42d4e838b5cf89bb466b0d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af2f174ce42d4e838b5cf89bb466b0d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e9e0e2eb54a4149bd265c6cc164f974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26d44557b7464a37bf2870582072aa2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39097 } health_report { overall_health: UNKNOWN } } }
I20260929 16:30:14.076982 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:14.073315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45590 (local address 127.0.98.1:34929)
0929 16:30:14.073447 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:14.073450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:14.073586 (+ 136us) server_negotiation.cc:407] Connection header received
0929 16:30:14.073765 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:14.073767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:14.073823 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:14.073913 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:30:14.074103 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:14.074609 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:14.075474 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:14.075751 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:14.075845 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:14.076122 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:30:14.076592 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:30:14.076686 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:14.076819 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 16:30:14.076867 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:30:14.083021 20529 raft_consensus.cc:1261] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d [term 1 FOLLOWER]: Refusing update from remote peer af2f174ce42d4e838b5cf89bb466b0d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:14.083058 20398 raft_consensus.cc:1261] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974 [term 1 FOLLOWER]: Refusing update from remote peer af2f174ce42d4e838b5cf89bb466b0d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:14.083698 20600 consensus_queue.cc:1035] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d44557b7464a37bf2870582072aa2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:14.083932 20605 consensus_queue.cc:1035] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e9e0e2eb54a4149bd265c6cc164f974" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:14.095536 20611 mvcc.cc:205] Tried to move back new op lower bound from 7334704800075997184 to 7334704799717179392. Current Snapshot: MvccSnapshot[applied={T|T < 7334704800075997184}]
I20260929 16:30:14.101665 20122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:30:14.101855 20122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:30:14.104722 20122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7c40fceec864a978618b95f01ff3598: Sending DeleteTablet for 3 replicas of tablet 9139f9c8e0b849849f8586851d343f2b
I20260929 16:30:14.105685 20247 tablet_service.cc:1514] Processing DeleteTablet for tablet 9139f9c8e0b849849f8586851d343f2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:14 UTC) from {username='slave'} at 127.0.0.1:45582
I20260929 16:30:14.105865 20509 tablet_service.cc:1514] Processing DeleteTablet for tablet 9139f9c8e0b849849f8586851d343f2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:14 UTC) from {username='slave'} at 127.0.0.1:37792
I20260929 16:30:14.106330 20378 tablet_service.cc:1514] Processing DeleteTablet for tablet 9139f9c8e0b849849f8586851d343f2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:30:14 UTC) from {username='slave'} at 127.0.0.1:42952
I20260929 16:30:14.106637 20620 tablet_replica.cc:330] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5: stopping tablet replica
I20260929 16:30:14.106745 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 20185
I20260929 16:30:14.107061 20620 raft_consensus.cc:2229] T 9139f9c8e0b849849f8586851d343f2b P af2f174ce42d4e838b5cf89bb466b0d5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:30:14.108904 20621 tablet_replica.cc:330] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d: stopping tablet replica
I20260929 16:30:14.109304 20621 raft_consensus.cc:2229] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:30:14.109644 20621 raft_consensus.cc:2258] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:14.110069 20622 tablet_replica.cc:330] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974: stopping tablet replica
I20260929 16:30:14.110402 20622 raft_consensus.cc:2229] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:30:14.110757 20622 raft_consensus.cc:2258] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:30:14.111297 20621 ts_tablet_manager.cc:1905] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:30:14.112100 20622 ts_tablet_manager.cc:1905] T 9139f9c8e0b849849f8586851d343f2b P 6e9e0e2eb54a4149bd265c6cc164f974: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:30:14.114334 20621 ts_tablet_manager.cc:1918] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:30:14.114440 20621 log.cc:1196] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d: Deleting WAL directory at /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790699152396020-392-0/minicluster-data/ts-2/wal/wals/9139f9c8e0b849849f8586851d343f2b
W20260929 16:30:14.114635 20112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34929 (error 108)
I20260929 16:30:14.114769 20621 ts_tablet_manager.cc:1939] T 9139f9c8e0b849849f8586851d343f2b P 26d44557b7464a37bf2870582072aa2d: Deleting consensus metadata
I20260929 16:30:14.114846 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 20316
I20260929 16:30:14.116137 20110 catalog_manager.cc:4841] TS 26d44557b7464a37bf2870582072aa2d (127.0.98.3:39097): tablet 9139f9c8e0b849849f8586851d343f2b (table test-table [id=70f3ee017a7640c5ba9c17446067fac7]) successfully deleted
I20260929 16:30:14.117682 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:14.117292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34929 (local address 127.0.0.1:45602)
0929 16:30:14.117528 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:14.117595 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":90,"threads_started":1}
W20260929 16:30:14.117929 20112 catalog_manager.cc:4584] TS af2f174ce42d4e838b5cf89bb466b0d5 (127.0.98.1:34929): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9139f9c8e0b849849f8586851d343f2b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34929: connect: Connection refused (error 111)
I20260929 16:30:14.121654 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 20447
I20260929 16:30:14.123135 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:14.121915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42391 (local address 127.0.0.1:42958)
0929 16:30:14.122930 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:14.123001 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":899}
I20260929 16:30:14.127305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 20094
2026-09-29T16:30:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1567 ms)
[----------] 6 tests from EncryptionPolicyTest (11783 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:14 dist-test-slave-vq9v krb5kdc[20627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:14 dist-test-slave-vq9v krb5kdc[20627](info): set up 2 sockets
Sep 29 16:30:14 dist-test-slave-vq9v krb5kdc[20627](info): commencing operation
krb5kdc: starting...
W20260929 16:30:15.181524 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:15 dist-test-slave-vq9v krb5kdc[20627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:30:15.312407 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34823
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:15.425930 20640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:15.426302 20640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:15.426393 20640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:15.430676 20640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:15.430797 20640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:15.430836 20640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:15.430877 20640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:15.430910 20640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:15.436745 20640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34823
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:15.438024 20640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:15.439129 20640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:15.447949 20648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:15.447933 20646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:15.448064 20640 server_base.cc:1034] running on GCE node
W20260929 16:30:15.447933 20645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:15.449004 20640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:15.450261 20640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:15.451447 20640 hybrid_clock.cc:648] HybridClock initialized: now 1790699415451428 us; error 42 us; skew 500 ppm
Sep 29 16:30:15 dist-test-slave-vq9v krb5kdc[20627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:15.454654 20640 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:30:15.455724 20640 webserver.cc:460] Webserver started at http://127.0.0.1:40481/ using document root <none> and password file <none>
I20260929 16:30:15.456321 20640 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:15.456421 20640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:15.456676 20640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:15.458912 20640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "18c5bbb509b64b3dbda1fe8f018b8bfd"
format_stamp: "Formatted at 2026-09-29 16:30:15 on dist-test-slave-vq9v"
server_key: "7d977180e2301a1c34e63d05db75cde9"
server_key_iv: "41b60303bdac352a9f7554f70faa893d"
server_key_version: "encryptionkey@0"
I20260929 16:30:15.459527 20640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "18c5bbb509b64b3dbda1fe8f018b8bfd"
format_stamp: "Formatted at 2026-09-29 16:30:15 on dist-test-slave-vq9v"
server_key: "7d977180e2301a1c34e63d05db75cde9"
server_key_iv: "41b60303bdac352a9f7554f70faa893d"
server_key_version: "encryptionkey@0"
I20260929 16:30:15.463547 20640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:30:15.466346 20655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:15.467728 20640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:15.467895 20640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "18c5bbb509b64b3dbda1fe8f018b8bfd"
format_stamp: "Formatted at 2026-09-29 16:30:15 on dist-test-slave-vq9v"
server_key: "7d977180e2301a1c34e63d05db75cde9"
server_key_iv: "41b60303bdac352a9f7554f70faa893d"
server_key_version: "encryptionkey@0"
I20260929 16:30:15.468032 20640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:15.491631 20640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:15.495873 20640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:15.496155 20640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:15.504376 20640 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34823
I20260929 16:30:15.504406 20707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34823 every 8 connection(s)
I20260929 16:30:15.505506 20640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:15.508471 20708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:15.509840 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 20640
I20260929 16:30:15.510015 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:15.510357 392 external_mini_cluster.cc:1428] Setting key 57bd5baac81a30361ecc172ff15fe7c3
I20260929 16:30:15.515678 20708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd: Bootstrap starting.
I20260929 16:30:15.518599 20708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:15.519606 20708 log.cc:824] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:30:15 dist-test-slave-vq9v krb5kdc[20627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:30:15.522290 20708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd: No bootstrap required, opened a new log
I20260929 16:30:15.526983 20708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c5bbb509b64b3dbda1fe8f018b8bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34823 } }
I20260929 16:30:15.527226 20708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:15.527262 20708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c5bbb509b64b3dbda1fe8f018b8bfd, State: Initialized, Role: FOLLOWER
I20260929 16:30:15.527271 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:15.512073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41441 (local address 127.0.0.1:34823)
0929 16:30:15.512579 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:15.512602 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:15.512652 (+ 50us) server_negotiation.cc:407] Connection header received
0929 16:30:15.513570 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:15.513597 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:15.514068 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:15.514510 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:15.515419 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.516848 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.517893 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.518608 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.518783 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:15.522029 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:15.522034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:15.522050 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:15.522080 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:15.523884 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:15.524590 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:15.524595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:15.524601 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:15.524724 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:15.525127 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:15.525129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:15.525131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:15.525609 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:15.525794 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:15.526143 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 16:30:15.526410 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":158,"threads_started":1}
I20260929 16:30:15.527815 20708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c5bbb509b64b3dbda1fe8f018b8bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34823 } }
I20260929 16:30:15.527978 20708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:15.528070 20708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:15.528200 20708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:15.529517 20708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c5bbb509b64b3dbda1fe8f018b8bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34823 } }
I20260929 16:30:15.529906 20708 leader_election.cc:302] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18c5bbb509b64b3dbda1fe8f018b8bfd; no voters:
I20260929 16:30:15.530319 20708 leader_election.cc:288] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:15.530432 20713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:15.530751 20713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd [term 1 LEADER]: Becoming Leader. State: Replica: 18c5bbb509b64b3dbda1fe8f018b8bfd, State: Running, Role: LEADER
I20260929 16:30:15.530790 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:30:15.531080 20713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c5bbb509b64b3dbda1fe8f018b8bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34823 } }
I20260929 16:30:15.531781 20708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:15.532228 20713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18c5bbb509b64b3dbda1fe8f018b8bfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c5bbb509b64b3dbda1fe8f018b8bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34823 } } }
I20260929 16:30:15.532429 20713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:15.532698 20713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18c5bbb509b64b3dbda1fe8f018b8bfd. Latest consensus state: current_term: 1 leader_uuid: "18c5bbb509b64b3dbda1fe8f018b8bfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c5bbb509b64b3dbda1fe8f018b8bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34823 } } }
I20260929 16:30:15.532856 20713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:15.533195 20723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:15.536592 20723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:15.543735 20723 catalog_manager.cc:1347] Generated new cluster ID: cbb6c6ea8b1c4f3e9904c37b637284fd
I20260929 16:30:15.543807 20723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:15.543969 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:15.534354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48833 (local address 127.0.0.1:34823)
0929 16:30:15.534481 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:15.534486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:15.534794 (+ 308us) server_negotiation.cc:407] Connection header received
0929 16:30:15.534930 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:15.534933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:15.534983 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:15.535076 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:15.535893 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.536711 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.537736 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:15.537987 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:15.538106 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:15.539902 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:15.539906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:15.539909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:15.539936 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:15.541771 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:15.542331 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:15.542334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:15.542336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:15.542425 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:15.542757 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:15.542761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:15.542762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:15.542953 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:15.543173 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:15.543687 (+ 514us) server_negotiation.cc:299] Negotiation successful
0929 16:30:15.543807 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:30:15.565999 20723 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:15.567034 20723 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:15.587044 20723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18c5bbb509b64b3dbda1fe8f018b8bfd: Generated new TSK 0
I20260929 16:30:15.588063 20723 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:15.603223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 20640
W20260929 16:30:15.608278 20625 connection.cc:535] client connection to 127.0.0.1:34823 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34823): Connection reset by peer (error 104)
2026-09-29T16:30:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1473 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:15 dist-test-slave-vq9v krb5kdc[20739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:15 dist-test-slave-vq9v krb5kdc[20739](info): set up 2 sockets
Sep 29 16:30:15 dist-test-slave-vq9v krb5kdc[20739](info): commencing operation
krb5kdc: starting...
W20260929 16:30:16.717612 392 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:16 dist-test-slave-vq9v krb5kdc[20739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:30:16.844758 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:16.959455 20752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:16.959767 20752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:16.959811 20752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:16.964146 20752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:16.964233 20752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:16.964253 20752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:16.964270 20752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:16.964286 20752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:16.970093 20752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33653
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:16.971266 20752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:16.972342 20752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:16.981312 20757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:16.981315 20758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:16.981317 20760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:16.981402 20752 server_base.cc:1034] running on GCE node
I20260929 16:30:16.982245 20752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:16.983475 20752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:16.984683 20752 hybrid_clock.cc:648] HybridClock initialized: now 1790699416984670 us; error 36 us; skew 500 ppm
Sep 29 16:30:16 dist-test-slave-vq9v krb5kdc[20739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:16.987743 20752 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:30:16.988731 20752 webserver.cc:460] Webserver started at http://127.0.0.1:36879/ using document root <none> and password file <none>
I20260929 16:30:16.989389 20752 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:16.989488 20752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:16.989743 20752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:16.991912 20752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "9ace4fdd5a3d4edb8cc4becf7c2d853e"
format_stamp: "Formatted at 2026-09-29 16:30:16 on dist-test-slave-vq9v"
server_key: "40430f731bdcc5368c1d43ef08030f15"
server_key_iv: "89722905aeb85d405caeddb3511d37b8"
server_key_version: "encryptionkey@0"
I20260929 16:30:16.992516 20752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "9ace4fdd5a3d4edb8cc4becf7c2d853e"
format_stamp: "Formatted at 2026-09-29 16:30:16 on dist-test-slave-vq9v"
server_key: "40430f731bdcc5368c1d43ef08030f15"
server_key_iv: "89722905aeb85d405caeddb3511d37b8"
server_key_version: "encryptionkey@0"
I20260929 16:30:16.996393 20752 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:30:16.998950 20767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:17.000156 20752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:30:17.000324 20752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "9ace4fdd5a3d4edb8cc4becf7c2d853e"
format_stamp: "Formatted at 2026-09-29 16:30:16 on dist-test-slave-vq9v"
server_key: "40430f731bdcc5368c1d43ef08030f15"
server_key_iv: "89722905aeb85d405caeddb3511d37b8"
server_key_version: "encryptionkey@0"
I20260929 16:30:17.000450 20752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:17.021970 20752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:17.025511 20752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:17.025748 20752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:17.033190 20752 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33653
I20260929 16:30:17.033260 20819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33653 every 8 connection(s)
I20260929 16:30:17.034230 20752 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:17.037084 20820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:17.042114 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 20752
I20260929 16:30:17.042251 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:17.042599 392 external_mini_cluster.cc:1428] Setting key 6a69255931f6ef1ca63769c52229253f
I20260929 16:30:17.042953 20820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e: Bootstrap starting.
I20260929 16:30:17.045768 20820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:17.046681 20820 log.cc:824] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:17.048931 20820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e: No bootstrap required, opened a new log
I20260929 16:30:17.053334 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:17.044180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36377 (local address 127.0.0.1:33653)
0929 16:30:17.044662 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:17.044681 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:17.044724 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:30:17.045626 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:17.045651 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:17.046127 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:17.046518 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:17.046966 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:17.048344 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:17.049381 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:17.050047 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:17.050225 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:17.051114 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:17.051120 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:17.051137 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:17.051658 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:17.051731 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:17.051762 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:17.051919 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:17.052114 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 16:30:17.052294 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":138,"threads_started":1}
I20260929 16:30:17.053676 20820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace4fdd5a3d4edb8cc4becf7c2d853e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33653 } }
I20260929 16:30:17.053957 20820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:17.054036 20820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ace4fdd5a3d4edb8cc4becf7c2d853e, State: Initialized, Role: FOLLOWER
I20260929 16:30:17.054512 20820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace4fdd5a3d4edb8cc4becf7c2d853e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33653 } }
I20260929 16:30:17.054682 20820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:17.054749 20820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:17.054834 20820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:17.056047 20820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace4fdd5a3d4edb8cc4becf7c2d853e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33653 } }
I20260929 16:30:17.056417 20820 leader_election.cc:302] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ace4fdd5a3d4edb8cc4becf7c2d853e; no voters:
I20260929 16:30:17.056775 20820 leader_election.cc:288] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:17.056789 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:30:17.056843 20825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:17.057056 20825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e [term 1 LEADER]: Becoming Leader. State: Replica: 9ace4fdd5a3d4edb8cc4becf7c2d853e, State: Running, Role: LEADER
I20260929 16:30:17.057435 20825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace4fdd5a3d4edb8cc4becf7c2d853e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33653 } }
I20260929 16:30:17.058112 20820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:17.058852 20827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ace4fdd5a3d4edb8cc4becf7c2d853e. Latest consensus state: current_term: 1 leader_uuid: "9ace4fdd5a3d4edb8cc4becf7c2d853e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace4fdd5a3d4edb8cc4becf7c2d853e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33653 } } }
I20260929 16:30:17.059056 20826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ace4fdd5a3d4edb8cc4becf7c2d853e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace4fdd5a3d4edb8cc4becf7c2d853e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33653 } } }
I20260929 16:30:17.059151 20826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:17.059036 20827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:17.059691 20838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:17.063021 20838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:17.065266 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:17.059468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38945 (local address 127.0.0.1:33653)
0929 16:30:17.059571 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:17.059576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:17.060611 (+ 1035us) server_negotiation.cc:407] Connection header received
0929 16:30:17.060762 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:17.060765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:17.060813 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:17.060905 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:17.061242 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:17.061989 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:17.063137 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:17.063452 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:17.063570 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:17.064710 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:17.064713 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:17.064716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:17.064731 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:17.064760 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:17.064774 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:17.064874 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:17.065044 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 16:30:17.065085 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 16:30:17.069646 20838 catalog_manager.cc:1347] Generated new cluster ID: a50c3b88f0dd4ed290a7214b8035130b
I20260929 16:30:17.069729 20838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:17.085709 20838 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:17.086844 20838 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:17.092917 20838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ace4fdd5a3d4edb8cc4becf7c2d853e: Generated new TSK 0
I20260929 16:30:17.093757 20838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:17.101945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 20752
W20260929 16:30:17.107519 20737 connection.cc:535] client connection to 127.0.0.1:33653 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33653): Connection reset by peer (error 104)
2026-09-29T16:30:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1499 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:17 dist-test-slave-vq9v krb5kdc[20851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:17 dist-test-slave-vq9v krb5kdc[20851](info): set up 2 sockets
Sep 29 16:30:17 dist-test-slave-vq9v krb5kdc[20851](info): commencing operation
krb5kdc: starting...
W20260929 16:30:18.153517 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:18 dist-test-slave-vq9v krb5kdc[20851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:30:18.289019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38021
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:18.407910 20864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:18.408227 20864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:18.408272 20864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:18.412452 20864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:18.412539 20864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:18.412560 20864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:18.412577 20864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:18.412592 20864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:18.418582 20864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38021
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:18.419788 20864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:18.420905 20864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:18.431167 20869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:18.431198 20870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:18.431173 20872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:18.431453 20864 server_base.cc:1034] running on GCE node
I20260929 16:30:18.432101 20864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:18.433610 20864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:18.434851 20864 hybrid_clock.cc:648] HybridClock initialized: now 1790699418434818 us; error 68 us; skew 500 ppm
Sep 29 16:30:18 dist-test-slave-vq9v krb5kdc[20851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:18.438323 20864 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:30:18.439438 20864 webserver.cc:460] Webserver started at http://127.0.0.1:36579/ using document root <none> and password file <none>
I20260929 16:30:18.440049 20864 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:18.440150 20864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:18.440399 20864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:18.442596 20864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "ec420c643e0244a5affd7d1aef74a9ee"
format_stamp: "Formatted at 2026-09-29 16:30:18 on dist-test-slave-vq9v"
server_key: "41907699437ba7894ec3abd0bad1a9fc"
server_key_iv: "43904e4694ff8c31c1091f56d061a95e"
server_key_version: "encryptionkey@0"
I20260929 16:30:18.443192 20864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "ec420c643e0244a5affd7d1aef74a9ee"
format_stamp: "Formatted at 2026-09-29 16:30:18 on dist-test-slave-vq9v"
server_key: "41907699437ba7894ec3abd0bad1a9fc"
server_key_iv: "43904e4694ff8c31c1091f56d061a95e"
server_key_version: "encryptionkey@0"
I20260929 16:30:18.447284 20864 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:30:18.450058 20879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:18.451418 20864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:18.451586 20864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "ec420c643e0244a5affd7d1aef74a9ee"
format_stamp: "Formatted at 2026-09-29 16:30:18 on dist-test-slave-vq9v"
server_key: "41907699437ba7894ec3abd0bad1a9fc"
server_key_iv: "43904e4694ff8c31c1091f56d061a95e"
server_key_version: "encryptionkey@0"
I20260929 16:30:18.451722 20864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:18.456789 20864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:18.460085 20864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:18.460299 20864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:18.467562 20864 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38021
I20260929 16:30:18.467587 20931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38021 every 8 connection(s)
I20260929 16:30:18.468516 20864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:18.471293 20932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:18.476104 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 20864
I20260929 16:30:18.476274 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:18.476557 392 external_mini_cluster.cc:1428] Setting key 6bba5cb369518da364e981fa90fb83d6
I20260929 16:30:18.476953 20932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee: Bootstrap starting.
I20260929 16:30:18.479449 20932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:18.480253 20932 log.cc:824] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:18.481920 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:18.478014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42635 (local address 127.0.0.1:38021)
0929 16:30:18.478484 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:18.478498 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:18.478532 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:30:18.479159 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:18.479177 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:18.479474 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:18.479846 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:18.480661 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:18.480665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:18.480678 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:18.480688 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:18.480743 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:18.480763 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:18.480882 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:18.480990 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 16:30:18.481155 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":149,"threads_started":1}
I20260929 16:30:18.482381 20932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee: No bootstrap required, opened a new log
I20260929 16:30:18.484743 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:30:18.486497 20932 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec420c643e0244a5affd7d1aef74a9ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38021 } }
I20260929 16:30:18.486796 20932 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:18.486866 20932 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec420c643e0244a5affd7d1aef74a9ee, State: Initialized, Role: FOLLOWER
I20260929 16:30:18.487421 20932 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec420c643e0244a5affd7d1aef74a9ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38021 } }
I20260929 16:30:18.487597 20932 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:18.487677 20932 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:18.487809 20932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:18.489001 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:18.487159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53447 (local address 127.0.0.1:38021)
0929 16:30:18.487278 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:18.487287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:18.487505 (+ 218us) server_negotiation.cc:407] Connection header received
0929 16:30:18.487692 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:18.487695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:18.487738 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:18.487830 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:18.488551 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:18.488554 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:18.488556 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:18.488566 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:18.488592 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:18.488602 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:18.488672 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:18.488804 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 16:30:18.488839 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 16:30:18.489163 20932 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec420c643e0244a5affd7d1aef74a9ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38021 } }
I20260929 16:30:18.489554 20932 leader_election.cc:302] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec420c643e0244a5affd7d1aef74a9ee; no voters:
I20260929 16:30:18.489818 20932 leader_election.cc:288] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:18.489926 20942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:18.490191 20942 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee [term 1 LEADER]: Becoming Leader. State: Replica: ec420c643e0244a5affd7d1aef74a9ee, State: Running, Role: LEADER
I20260929 16:30:18.490550 20942 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec420c643e0244a5affd7d1aef74a9ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38021 } }
I20260929 16:30:18.491122 20932 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:18.492205 20944 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec420c643e0244a5affd7d1aef74a9ee. Latest consensus state: current_term: 1 leader_uuid: "ec420c643e0244a5affd7d1aef74a9ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec420c643e0244a5affd7d1aef74a9ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38021 } } }
I20260929 16:30:18.492331 20944 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:18.492494 20943 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec420c643e0244a5affd7d1aef74a9ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec420c643e0244a5affd7d1aef74a9ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38021 } } }
I20260929 16:30:18.492673 20943 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:18.492957 20951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:18.496225 20951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:18.502192 20951 catalog_manager.cc:1347] Generated new cluster ID: 993f1731c5764b928e9950b4d86ac1c5
I20260929 16:30:18.502303 20951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:18.523663 20951 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:18.524116 20951 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:18.532635 20951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec420c643e0244a5affd7d1aef74a9ee: Generated new TSK 0
I20260929 16:30:18.533707 20951 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:18.545157 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 20864
2026-09-29T16:30:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1446 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:18 dist-test-slave-vq9v krb5kdc[20963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:18 dist-test-slave-vq9v krb5kdc[20963](info): set up 2 sockets
Sep 29 16:30:18 dist-test-slave-vq9v krb5kdc[20963](info): commencing operation
krb5kdc: starting...
W20260929 16:30:19.601682 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:19 dist-test-slave-vq9v krb5kdc[20963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:30:19.728807 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32971
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:19.842370 20976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:19.842751 20976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:19.842844 20976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:19.847102 20976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:19.847220 20976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:19.847260 20976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:19.847301 20976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:19.847342 20976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:19.853024 20976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:19.854238 20976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:19.855379 20976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:30:19.864473 20976 server_base.cc:1034] running on GCE node
W20260929 16:30:19.864446 20981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:19.864496 20982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:19.864493 20984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:19.865352 20976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:19.866447 20976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:19.867622 20976 hybrid_clock.cc:648] HybridClock initialized: now 1790699419867597 us; error 41 us; skew 500 ppm
Sep 29 16:30:19 dist-test-slave-vq9v krb5kdc[20963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:19.879298 20976 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:30:19.880367 20976 webserver.cc:460] Webserver started at http://127.0.0.1:36977/ using document root <none> and password file <none>
I20260929 16:30:19.880966 20976 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:19.881064 20976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:19.881377 20976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:19.883533 20976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "d4f88e4ebfe1481397cab7b1bb1dcfa9"
format_stamp: "Formatted at 2026-09-29 16:30:19 on dist-test-slave-vq9v"
server_key: "80a4150d53abf0167dd3adc8642b6e65"
server_key_iv: "78a82345191d4e5dbdd4ae70bb3382ad"
server_key_version: "encryptionkey@0"
I20260929 16:30:19.884131 20976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "d4f88e4ebfe1481397cab7b1bb1dcfa9"
format_stamp: "Formatted at 2026-09-29 16:30:19 on dist-test-slave-vq9v"
server_key: "80a4150d53abf0167dd3adc8642b6e65"
server_key_iv: "78a82345191d4e5dbdd4ae70bb3382ad"
server_key_version: "encryptionkey@0"
I20260929 16:30:19.888043 20976 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:30:19.890643 20991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:19.891956 20976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:30:19.892112 20976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "d4f88e4ebfe1481397cab7b1bb1dcfa9"
format_stamp: "Formatted at 2026-09-29 16:30:19 on dist-test-slave-vq9v"
server_key: "80a4150d53abf0167dd3adc8642b6e65"
server_key_iv: "78a82345191d4e5dbdd4ae70bb3382ad"
server_key_version: "encryptionkey@0"
I20260929 16:30:19.892244 20976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:19.927135 20976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:19.930215 20976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:19.930446 20976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:19.937793 20976 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32971
I20260929 16:30:19.937820 21043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32971 every 8 connection(s)
I20260929 16:30:19.938766 20976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:19.941592 21044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:19.946467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 20976
I20260929 16:30:19.946610 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:19.946926 392 external_mini_cluster.cc:1428] Setting key aa8e3f277981da3c57f987e24e01444f
I20260929 16:30:19.947645 21044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9: Bootstrap starting.
I20260929 16:30:19.950102 21044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:19.951048 21044 log.cc:824] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:19.953334 21044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9: No bootstrap required, opened a new log
Sep 29 16:30:19 dist-test-slave-vq9v krb5kdc[20963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:30:19.957734 21044 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f88e4ebfe1481397cab7b1bb1dcfa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32971 } }
I20260929 16:30:19.957967 21044 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:19.958051 21044 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4f88e4ebfe1481397cab7b1bb1dcfa9, State: Initialized, Role: FOLLOWER
I20260929 16:30:19.958537 21044 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f88e4ebfe1481397cab7b1bb1dcfa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32971 } }
I20260929 16:30:19.958663 21044 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:19.958752 21044 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:19.958864 21044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:19.959805 21044 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f88e4ebfe1481397cab7b1bb1dcfa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32971 } }
I20260929 16:30:19.960168 21044 leader_election.cc:302] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4f88e4ebfe1481397cab7b1bb1dcfa9; no voters:
I20260929 16:30:19.960525 21044 leader_election.cc:288] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:19.960623 21049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:19.960827 21049 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9 [term 1 LEADER]: Becoming Leader. State: Replica: d4f88e4ebfe1481397cab7b1bb1dcfa9, State: Running, Role: LEADER
I20260929 16:30:19.961135 21049 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f88e4ebfe1481397cab7b1bb1dcfa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32971 } }
I20260929 16:30:19.961995 21044 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:19.962252 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:19.948284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36783 (local address 127.0.0.1:32971)
0929 16:30:19.948681 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:19.948697 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:19.948731 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:30:19.949427 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:19.949445 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:19.949787 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:19.950067 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:19.950969 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:19.951989 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:19.953165 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:19.953936 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:19.954117 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:19.956886 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:19.956892 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:19.956907 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:19.956935 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:19.959036 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:19.959478 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:19.959484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:19.959490 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:19.959614 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:19.959918 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:19.959921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:19.959923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:19.960407 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:19.960604 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:19.960998 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 16:30:19.961438 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":125,"threads_started":1}
I20260929 16:30:19.962390 21049 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4f88e4ebfe1481397cab7b1bb1dcfa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f88e4ebfe1481397cab7b1bb1dcfa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32971 } } }
I20260929 16:30:19.962591 21051 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4f88e4ebfe1481397cab7b1bb1dcfa9. Latest consensus state: current_term: 1 leader_uuid: "d4f88e4ebfe1481397cab7b1bb1dcfa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f88e4ebfe1481397cab7b1bb1dcfa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32971 } } }
I20260929 16:30:19.962656 21049 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:19.962688 21051 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:19.963069 21054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:19.965873 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:30:19.966931 21054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:19.972710 21054 catalog_manager.cc:1347] Generated new cluster ID: 01bd6c7934bf49d1aae9cb0ccfc25192
I20260929 16:30:19.972777 21054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:19.980134 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:19.970851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44687 (local address 127.0.0.1:32971)
0929 16:30:19.970988 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:19.970992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:19.971006 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:30:19.971044 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:19.971047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:19.971090 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:19.971197 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:19.972449 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:19.973295 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:19.974522 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:19.974877 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:19.974972 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:19.976617 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:19.976620 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:19.976623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:19.976653 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:19.978398 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:19.978902 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:19.978905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:19.978907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:19.978989 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:19.979266 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:19.979269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:19.979271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:19.979473 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:19.979585 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:19.979854 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 16:30:19.979971 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:30:19.984184 21054 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:19.985560 21054 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:19.992856 21054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4f88e4ebfe1481397cab7b1bb1dcfa9: Generated new TSK 0
I20260929 16:30:19.993762 21054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:20.004549 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 20976
2026-09-29T16:30:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1455 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:20 dist-test-slave-vq9v krb5kdc[21075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:20 dist-test-slave-vq9v krb5kdc[21075](info): set up 2 sockets
Sep 29 16:30:20 dist-test-slave-vq9v krb5kdc[21075](info): commencing operation
krb5kdc: starting...
W20260929 16:30:21.065647 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:21 dist-test-slave-vq9v krb5kdc[21075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:30:21.196746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35953
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:21.322649 21088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:21.322937 21088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:21.322981 21088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:21.327193 21088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:21.327276 21088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:21.327296 21088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:21.327313 21088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:21.327329 21088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:21.332867 21088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:21.334061 21088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:21.335098 21088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:21.344338 21096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:21.344385 21088 server_base.cc:1034] running on GCE node
W20260929 16:30:21.344338 21094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:21.344338 21093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:21.345314 21088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:21.346539 21088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:21.347750 21088 hybrid_clock.cc:648] HybridClock initialized: now 1790699421347715 us; error 54 us; skew 500 ppm
Sep 29 16:30:21 dist-test-slave-vq9v krb5kdc[21075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:21.350728 21088 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:30:21.351744 21088 webserver.cc:460] Webserver started at http://127.0.0.1:38377/ using document root <none> and password file <none>
I20260929 16:30:21.352332 21088 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:21.352429 21088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:21.352674 21088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:21.354879 21088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "c37a54ead344451fb08e7fd5a0768cc0"
format_stamp: "Formatted at 2026-09-29 16:30:21 on dist-test-slave-vq9v"
server_key: "f53fb4f7d37b66e68a06307a2653ea04"
server_key_iv: "0dc61f4a79be8aff4125a362a4c3c9f3"
server_key_version: "encryptionkey@0"
I20260929 16:30:21.355451 21088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "c37a54ead344451fb08e7fd5a0768cc0"
format_stamp: "Formatted at 2026-09-29 16:30:21 on dist-test-slave-vq9v"
server_key: "f53fb4f7d37b66e68a06307a2653ea04"
server_key_iv: "0dc61f4a79be8aff4125a362a4c3c9f3"
server_key_version: "encryptionkey@0"
I20260929 16:30:21.359318 21088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:30:21.361882 21103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:21.363267 21088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:21.363443 21088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "c37a54ead344451fb08e7fd5a0768cc0"
format_stamp: "Formatted at 2026-09-29 16:30:21 on dist-test-slave-vq9v"
server_key: "f53fb4f7d37b66e68a06307a2653ea04"
server_key_iv: "0dc61f4a79be8aff4125a362a4c3c9f3"
server_key_version: "encryptionkey@0"
I20260929 16:30:21.363582 21088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:21.377794 21088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:21.381294 21088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:21.381533 21088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:21.389032 21088 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35953
I20260929 16:30:21.389050 21155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35953 every 8 connection(s)
I20260929 16:30:21.390184 21088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:21.393093 21156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:21.393923 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 21088
I20260929 16:30:21.394080 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:21.394423 392 external_mini_cluster.cc:1428] Setting key df159eddf9514ccca02c1a500c79c02e
I20260929 16:30:21.400288 21156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0: Bootstrap starting.
I20260929 16:30:21.403263 21156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:21.404244 21156 log.cc:824] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:21.404927 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:21.396005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50321 (local address 127.0.0.1:35953)
0929 16:30:21.396495 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:21.396512 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:21.396559 (+ 47us) server_negotiation.cc:407] Connection header received
0929 16:30:21.397350 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:21.397372 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:21.397781 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:21.398099 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:21.398535 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:21.399596 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:21.400859 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:21.401505 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:21.401714 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:21.402705 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:21.402711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:21.402726 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:21.403170 (+ 444us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:21.403323 (+ 153us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:21.403362 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:21.403498 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:21.403700 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 16:30:21.403940 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":158,"threads_started":1}
I20260929 16:30:21.406479 21156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0: No bootstrap required, opened a new log
I20260929 16:30:21.408126 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:30:21.410921 21156 raft_consensus.cc:348] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37a54ead344451fb08e7fd5a0768cc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35953 } }
I20260929 16:30:21.411298 21156 raft_consensus.cc:374] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:21.411377 21156 raft_consensus.cc:729] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37a54ead344451fb08e7fd5a0768cc0, State: Initialized, Role: FOLLOWER
I20260929 16:30:21.411868 21156 consensus_queue.cc:260] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37a54ead344451fb08e7fd5a0768cc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35953 } }
I20260929 16:30:21.411998 21156 raft_consensus.cc:388] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:21.412086 21156 raft_consensus.cc:482] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:21.412201 21156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:21.413298 21156 raft_consensus.cc:504] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37a54ead344451fb08e7fd5a0768cc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35953 } }
I20260929 16:30:21.413688 21156 leader_election.cc:302] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c37a54ead344451fb08e7fd5a0768cc0; no voters:
I20260929 16:30:21.414062 21156 leader_election.cc:288] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:21.414149 21166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:21.414367 21166 raft_consensus.cc:686] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0 [term 1 LEADER]: Becoming Leader. State: Replica: c37a54ead344451fb08e7fd5a0768cc0, State: Running, Role: LEADER
I20260929 16:30:21.414688 21166 consensus_queue.cc:237] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37a54ead344451fb08e7fd5a0768cc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35953 } }
I20260929 16:30:21.415351 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:21.410418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50347 (local address 127.0.0.1:35953)
0929 16:30:21.410558 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:21.410562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:21.410738 (+ 176us) server_negotiation.cc:407] Connection header received
0929 16:30:21.410849 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:21.410851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:21.410896 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:21.410975 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:21.411392 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:21.412253 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:21.413251 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:21.413617 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:21.413759 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:21.414770 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:21.414797 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:21.414803 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:21.414824 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:21.414859 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:21.414878 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:21.414937 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:21.415126 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 16:30:21.415199 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:30:21.415511 21156 sys_catalog.cc:564] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:21.415927 21166 sys_catalog.cc:455] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c37a54ead344451fb08e7fd5a0768cc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37a54ead344451fb08e7fd5a0768cc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35953 } } }
I20260929 16:30:21.416129 21166 sys_catalog.cc:458] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:21.416635 21171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:21.417120 21167 sys_catalog.cc:455] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c37a54ead344451fb08e7fd5a0768cc0. Latest consensus state: current_term: 1 leader_uuid: "c37a54ead344451fb08e7fd5a0768cc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37a54ead344451fb08e7fd5a0768cc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35953 } } }
I20260929 16:30:21.417349 21167 sys_catalog.cc:458] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:21.420246 21171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:21.426692 21171 catalog_manager.cc:1347] Generated new cluster ID: 07740101a765453fab76e073c08da479
I20260929 16:30:21.426792 21171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:21.452065 21171 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:21.453261 21171 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:21.460379 21171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c37a54ead344451fb08e7fd5a0768cc0: Generated new TSK 0
I20260929 16:30:21.461345 21171 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:21.471364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 21088
2026-09-29T16:30:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1466 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:21 dist-test-slave-vq9v krb5kdc[21187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:21 dist-test-slave-vq9v krb5kdc[21187](info): set up 2 sockets
Sep 29 16:30:21 dist-test-slave-vq9v krb5kdc[21187](info): commencing operation
krb5kdc: starting...
W20260929 16:30:22.530019 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:22 dist-test-slave-vq9v krb5kdc[21187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:30:22.664601 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:22.782361 21200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:22.782713 21200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:22.782800 21200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:22.787343 21200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:22.787457 21200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:22.787489 21200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:22.787521 21200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:22.787556 21200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:22.793335 21200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:22.794579 21200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:22.795684 21200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:22.805110 21206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:22.805287 21200 server_base.cc:1034] running on GCE node
W20260929 16:30:22.805104 21208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:22.805155 21205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:22.806093 21200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:22.807399 21200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:22.808609 21200 hybrid_clock.cc:648] HybridClock initialized: now 1790699422808585 us; error 35 us; skew 500 ppm
Sep 29 16:30:22 dist-test-slave-vq9v krb5kdc[21187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:22.811758 21200 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:30:22.812805 21200 webserver.cc:460] Webserver started at http://127.0.0.1:46631/ using document root <none> and password file <none>
I20260929 16:30:22.813495 21200 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:22.813599 21200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:22.813853 21200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:22.816336 21200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "e35ba339b3a543f5abe2e9db81f214b3"
format_stamp: "Formatted at 2026-09-29 16:30:22 on dist-test-slave-vq9v"
server_key: "eeb94b3fc485cc0529764fb866cbbbdd"
server_key_iv: "cf0889544c4eee40d93750449cb53996"
server_key_version: "encryptionkey@0"
I20260929 16:30:22.817024 21200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "e35ba339b3a543f5abe2e9db81f214b3"
format_stamp: "Formatted at 2026-09-29 16:30:22 on dist-test-slave-vq9v"
server_key: "eeb94b3fc485cc0529764fb866cbbbdd"
server_key_iv: "cf0889544c4eee40d93750449cb53996"
server_key_version: "encryptionkey@0"
I20260929 16:30:22.821426 21200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:30:22.824275 21215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:22.825771 21200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:22.825951 21200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "e35ba339b3a543f5abe2e9db81f214b3"
format_stamp: "Formatted at 2026-09-29 16:30:22 on dist-test-slave-vq9v"
server_key: "eeb94b3fc485cc0529764fb866cbbbdd"
server_key_iv: "cf0889544c4eee40d93750449cb53996"
server_key_version: "encryptionkey@0"
I20260929 16:30:22.826143 21200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:22.831820 21200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:22.835297 21200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:22.835521 21200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:22.843479 21200 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46653
I20260929 16:30:22.843504 21267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46653 every 8 connection(s)
I20260929 16:30:22.844470 21200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:22.847281 21268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:22.851907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 21200
I20260929 16:30:22.852027 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:22.852312 392 external_mini_cluster.cc:1428] Setting key c4936115eeafe62f035c65924ce191f7
I20260929 16:30:22.853399 21268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3: Bootstrap starting.
I20260929 16:30:22.855911 21268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:22.856835 21268 log.cc:824] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:22.858207 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:22.853836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33001 (local address 127.0.0.1:46653)
0929 16:30:22.854338 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:22.854353 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:22.854390 (+ 37us) server_negotiation.cc:407] Connection header received
0929 16:30:22.855073 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:22.855091 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:22.855406 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:22.855782 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:22.856696 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:22.856700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:22.856712 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:22.856723 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:22.856783 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:22.856802 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:22.856926 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:22.857074 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 16:30:22.857309 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":148,"threads_started":1}
I20260929 16:30:22.859218 21268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3: No bootstrap required, opened a new log
I20260929 16:30:22.861393 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:30:22.863338 21268 raft_consensus.cc:348] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35ba339b3a543f5abe2e9db81f214b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46653 } }
I20260929 16:30:22.863607 21268 raft_consensus.cc:374] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:22.863700 21268 raft_consensus.cc:729] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35ba339b3a543f5abe2e9db81f214b3, State: Initialized, Role: FOLLOWER
I20260929 16:30:22.864264 21268 consensus_queue.cc:260] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35ba339b3a543f5abe2e9db81f214b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46653 } }
I20260929 16:30:22.864437 21268 raft_consensus.cc:388] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:22.864538 21268 raft_consensus.cc:482] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:22.864673 21268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:22.865556 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:22.863722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54047 (local address 127.0.0.1:46653)
0929 16:30:22.863913 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:22.863918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:22.864023 (+ 105us) server_negotiation.cc:407] Connection header received
0929 16:30:22.864167 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:22.864170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:22.864208 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:22.864296 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:22.865090 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:22.865094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:22.865097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:22.865109 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:22.865148 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:22.865162 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:22.865283 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:22.865383 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 16:30:22.865423 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260929 16:30:22.866004 21268 raft_consensus.cc:504] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35ba339b3a543f5abe2e9db81f214b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46653 } }
I20260929 16:30:22.866379 21268 leader_election.cc:302] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e35ba339b3a543f5abe2e9db81f214b3; no voters:
I20260929 16:30:22.866746 21268 leader_election.cc:288] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:22.866863 21278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:22.867074 21278 raft_consensus.cc:686] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3 [term 1 LEADER]: Becoming Leader. State: Replica: e35ba339b3a543f5abe2e9db81f214b3, State: Running, Role: LEADER
I20260929 16:30:22.867443 21278 consensus_queue.cc:237] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35ba339b3a543f5abe2e9db81f214b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46653 } }
I20260929 16:30:22.868069 21268 sys_catalog.cc:564] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:22.869182 21279 sys_catalog.cc:455] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e35ba339b3a543f5abe2e9db81f214b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35ba339b3a543f5abe2e9db81f214b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46653 } } }
I20260929 16:30:22.869356 21279 sys_catalog.cc:458] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:22.869707 21280 sys_catalog.cc:455] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e35ba339b3a543f5abe2e9db81f214b3. Latest consensus state: current_term: 1 leader_uuid: "e35ba339b3a543f5abe2e9db81f214b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35ba339b3a543f5abe2e9db81f214b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46653 } } }
I20260929 16:30:22.869856 21280 sys_catalog.cc:458] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:22.869765 21287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:22.873278 21287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:22.878773 21287 catalog_manager.cc:1347] Generated new cluster ID: 99b5db73ce1c4ce79515b5a4bce7578d
I20260929 16:30:22.878849 21287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:22.886888 21287 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:22.887279 21287 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:22.894410 21287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e35ba339b3a543f5abe2e9db81f214b3: Generated new TSK 0
I20260929 16:30:22.895427 21287 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:22.906625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 21200
2026-09-29T16:30:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:22 dist-test-slave-vq9v krb5kdc[21299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:22 dist-test-slave-vq9v krb5kdc[21299](info): set up 2 sockets
Sep 29 16:30:22 dist-test-slave-vq9v krb5kdc[21299](info): commencing operation
krb5kdc: starting...
W20260929 16:30:24.045682 392 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:24 dist-test-slave-vq9v krb5kdc[21299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:30:24.177078 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42499
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:24.294906 21312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:24.295257 21312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:24.295346 21312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:24.299508 21312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:24.299629 21312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:24.299669 21312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:24.299710 21312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:24.299758 21312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:24.305366 21312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:24.306560 21312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:24.307687 21312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:30:24.316943 21312 server_base.cc:1034] running on GCE node
W20260929 16:30:24.316886 21320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:24.316880 21317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:24.316910 21318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:24.317827 21312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:24.318992 21312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:24.320235 21312 hybrid_clock.cc:648] HybridClock initialized: now 1790699424320210 us; error 39 us; skew 500 ppm
Sep 29 16:30:24 dist-test-slave-vq9v krb5kdc[21299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:24.323396 21312 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:30:24.324350 21312 webserver.cc:460] Webserver started at http://127.0.0.1:38427/ using document root <none> and password file <none>
I20260929 16:30:24.324952 21312 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:24.325047 21312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:24.325338 21312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:24.327530 21312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "a88b23d720c041ab9692bd89bc1b4764"
format_stamp: "Formatted at 2026-09-29 16:30:24 on dist-test-slave-vq9v"
server_key: "53b0f3b69fb825a68d8cb0bc043656cb"
server_key_iv: "489312ddbe838cf1b5660d9220dacd36"
server_key_version: "encryptionkey@0"
I20260929 16:30:24.328114 21312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "a88b23d720c041ab9692bd89bc1b4764"
format_stamp: "Formatted at 2026-09-29 16:30:24 on dist-test-slave-vq9v"
server_key: "53b0f3b69fb825a68d8cb0bc043656cb"
server_key_iv: "489312ddbe838cf1b5660d9220dacd36"
server_key_version: "encryptionkey@0"
I20260929 16:30:24.332007 21312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:30:24.334730 21327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:24.336020 21312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 16:30:24.336193 21312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "a88b23d720c041ab9692bd89bc1b4764"
format_stamp: "Formatted at 2026-09-29 16:30:24 on dist-test-slave-vq9v"
server_key: "53b0f3b69fb825a68d8cb0bc043656cb"
server_key_iv: "489312ddbe838cf1b5660d9220dacd36"
server_key_version: "encryptionkey@0"
I20260929 16:30:24.336326 21312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:24.349306 21312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:24.352892 21312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:24.353116 21312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:24.360597 21312 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42499
I20260929 16:30:24.360626 21379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42499 every 8 connection(s)
I20260929 16:30:24.361656 21312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:24.363909 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 21312
I20260929 16:30:24.364048 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:24.364323 392 external_mini_cluster.cc:1428] Setting key 799ad99cb5920f8ca7a69a962e1c7ce1
I20260929 16:30:24.364651 21380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:24.372112 21380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764: Bootstrap starting.
Sep 29 16:30:24 dist-test-slave-vq9v krb5kdc[21299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:30:24.374809 21380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:24.375751 21380 log.cc:824] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:24.378286 21380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764: No bootstrap required, opened a new log
I20260929 16:30:24.380499 21383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:24.365784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:37345 (local address 127.0.0.1:42499)
0929 16:30:24.366220 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:24.366238 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:24.366281 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:30:24.367043 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:24.367066 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:24.367560 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:24.367949 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:24.368622 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:24.370150 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:24.371064 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:24.371743 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:24.371922 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:24.374817 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:24.374826 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:24.374845 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:24.374878 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:24.376873 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:24.377505 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:24.377510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:24.377516 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:24.377659 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:24.377975 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:24.377978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:24.377980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:24.378486 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:24.378727 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:24.378957 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 16:30:24.379186 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":122,"threads_started":1}
I20260929 16:30:24.382287 21380 raft_consensus.cc:348] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a88b23d720c041ab9692bd89bc1b4764" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42499 } }
I20260929 16:30:24.382596 21380 raft_consensus.cc:374] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:24.382678 21380 raft_consensus.cc:729] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a88b23d720c041ab9692bd89bc1b4764, State: Initialized, Role: FOLLOWER
I20260929 16:30:24.383188 21380 consensus_queue.cc:260] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a88b23d720c041ab9692bd89bc1b4764" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42499 } }
I20260929 16:30:24.383368 21380 raft_consensus.cc:388] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:24.383438 21380 raft_consensus.cc:482] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:24.383538 21380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:24.383903 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:30:24.384603 21380 raft_consensus.cc:504] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a88b23d720c041ab9692bd89bc1b4764" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42499 } }
I20260929 16:30:24.384967 21380 leader_election.cc:302] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a88b23d720c041ab9692bd89bc1b4764; no voters:
I20260929 16:30:24.385430 21380 leader_election.cc:288] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:24.385514 21389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:24.385753 21389 raft_consensus.cc:686] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764 [term 1 LEADER]: Becoming Leader. State: Replica: a88b23d720c041ab9692bd89bc1b4764, State: Running, Role: LEADER
I20260929 16:30:24.386121 21389 consensus_queue.cc:237] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a88b23d720c041ab9692bd89bc1b4764" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42499 } }
I20260929 16:30:24.386956 21380 sys_catalog.cc:564] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:24.387421 21389 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a88b23d720c041ab9692bd89bc1b4764. Latest consensus state: current_term: 1 leader_uuid: "a88b23d720c041ab9692bd89bc1b4764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a88b23d720c041ab9692bd89bc1b4764" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42499 } } }
I20260929 16:30:24.387539 21389 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:24.387825 21390 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a88b23d720c041ab9692bd89bc1b4764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a88b23d720c041ab9692bd89bc1b4764" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42499 } } }
I20260929 16:30:24.387951 21390 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:24.389288 21394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:24.392383 21394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:24.396538 21383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:24.386154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:39545 (local address 127.0.0.1:42499)
0929 16:30:24.386364 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:24.386368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:24.386697 (+ 329us) server_negotiation.cc:407] Connection header received
0929 16:30:24.386802 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:24.386806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:24.386855 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:24.386944 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:24.388578 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:24.389382 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:24.390356 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:24.390686 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:24.390795 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:24.392561 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:24.392564 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:24.392567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:24.392591 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:24.394387 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:24.394960 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:24.394964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:24.394966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:24.395058 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:24.395433 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:24.395436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:24.395438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:24.395660 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:24.395798 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:24.396101 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 16:30:24.396198 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:30:24.398823 21394 catalog_manager.cc:1347] Generated new cluster ID: 46f122ad75ac4077959966c38066ee0a
I20260929 16:30:24.398896 21394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:24.423185 21394 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:24.424213 21394 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:24.430261 21394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a88b23d720c041ab9692bd89bc1b4764: Generated new TSK 0
I20260929 16:30:24.431139 21394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:24.443625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 21312
W20260929 16:30:24.448422 21297 connection.cc:535] client connection to 127.0.0.1:42499 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42499): Connection reset by peer (error 104)
2026-09-29T16:30:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1535 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:24 dist-test-slave-vq9v krb5kdc[21411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:24 dist-test-slave-vq9v krb5kdc[21411](info): set up 2 sockets
Sep 29 16:30:24 dist-test-slave-vq9v krb5kdc[21411](info): commencing operation
krb5kdc: starting...
W20260929 16:30:25.485603 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:25 dist-test-slave-vq9v krb5kdc[21411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:30:25.617331 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40293
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:25.733632 21424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:25.733987 21424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:25.734033 21424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:25.738366 21424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:25.738456 21424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:25.738476 21424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:25.738495 21424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:25.738511 21424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:25.744336 21424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:25.745509 21424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:25.746541 21424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:25.755800 21430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:25.755875 21424 server_base.cc:1034] running on GCE node
W20260929 16:30:25.755885 21429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:25.755820 21432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:25.756886 21424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:25.758237 21424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:25.759446 21424 hybrid_clock.cc:648] HybridClock initialized: now 1790699425759428 us; error 35 us; skew 500 ppm
Sep 29 16:30:25 dist-test-slave-vq9v krb5kdc[21411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:25.768798 21424 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:30:25.769982 21424 webserver.cc:460] Webserver started at http://127.0.0.1:38249/ using document root <none> and password file <none>
I20260929 16:30:25.770622 21424 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:25.770732 21424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:25.770978 21424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:25.773187 21424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "cf8e0e63787f49b589a54bd5f6ec4da2"
format_stamp: "Formatted at 2026-09-29 16:30:25 on dist-test-slave-vq9v"
server_key: "edcb41859ca69dd813b91f5ba7cdcf8a"
server_key_iv: "983c09c2bc858c493ec3c60a335c5339"
server_key_version: "encryptionkey@0"
I20260929 16:30:25.773892 21424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "cf8e0e63787f49b589a54bd5f6ec4da2"
format_stamp: "Formatted at 2026-09-29 16:30:25 on dist-test-slave-vq9v"
server_key: "edcb41859ca69dd813b91f5ba7cdcf8a"
server_key_iv: "983c09c2bc858c493ec3c60a335c5339"
server_key_version: "encryptionkey@0"
I20260929 16:30:25.778061 21424 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:30:25.780815 21439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:25.782313 21424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:30:25.782470 21424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "cf8e0e63787f49b589a54bd5f6ec4da2"
format_stamp: "Formatted at 2026-09-29 16:30:25 on dist-test-slave-vq9v"
server_key: "edcb41859ca69dd813b91f5ba7cdcf8a"
server_key_iv: "983c09c2bc858c493ec3c60a335c5339"
server_key_version: "encryptionkey@0"
I20260929 16:30:25.782608 21424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:25.793571 21424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:25.797003 21424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:25.797298 21424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:25.804649 21424 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40293
I20260929 16:30:25.804674 21491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40293 every 8 connection(s)
I20260929 16:30:25.805786 21424 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:25.808668 21492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:25.814563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 21424
I20260929 16:30:25.814738 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:25.814706 21492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2: Bootstrap starting.
I20260929 16:30:25.815045 392 external_mini_cluster.cc:1428] Setting key c7e16bafb68cb7f2399335718de7e5a0
I20260929 16:30:25.817741 21492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:25.818567 21492 log.cc:824] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:25.820622 21492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2: No bootstrap required, opened a new log
I20260929 16:30:25.824481 21492 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8e0e63787f49b589a54bd5f6ec4da2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40293 } }
I20260929 16:30:25.824697 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:25.816591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:40501 (local address 127.0.0.1:40293)
0929 16:30:25.817099 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:25.817121 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:25.817165 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:30:25.818000 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:25.818021 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:25.818402 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:25.818826 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:25.819247 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:25.820231 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:25.821275 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:25.821778 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:25.821982 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:25.822924 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:25.822930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:25.822945 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:25.823354 (+ 409us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:25.823412 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:25.823437 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:25.823581 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:25.823757 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 16:30:25.823909 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":138,"threads_started":1}
I20260929 16:30:25.824783 21492 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:25.824859 21492 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf8e0e63787f49b589a54bd5f6ec4da2, State: Initialized, Role: FOLLOWER
I20260929 16:30:25.825421 21492 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8e0e63787f49b589a54bd5f6ec4da2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40293 } }
I20260929 16:30:25.825591 21492 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:25.825695 21492 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:25.825817 21492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:25.826906 21492 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8e0e63787f49b589a54bd5f6ec4da2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40293 } }
I20260929 16:30:25.827291 21492 leader_election.cc:302] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf8e0e63787f49b589a54bd5f6ec4da2; no voters:
I20260929 16:30:25.827653 21492 leader_election.cc:288] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:25.827817 21497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:25.828028 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:30:25.828104 21497 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2 [term 1 LEADER]: Becoming Leader. State: Replica: cf8e0e63787f49b589a54bd5f6ec4da2, State: Running, Role: LEADER
I20260929 16:30:25.828471 21497 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8e0e63787f49b589a54bd5f6ec4da2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40293 } }
I20260929 16:30:25.829054 21492 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:25.830261 21498 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf8e0e63787f49b589a54bd5f6ec4da2. Latest consensus state: current_term: 1 leader_uuid: "cf8e0e63787f49b589a54bd5f6ec4da2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8e0e63787f49b589a54bd5f6ec4da2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40293 } } }
I20260929 16:30:25.830246 21499 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf8e0e63787f49b589a54bd5f6ec4da2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8e0e63787f49b589a54bd5f6ec4da2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40293 } } }
I20260929 16:30:25.830462 21499 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:25.830462 21498 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2 [sys.catalog]: This master's current role is: LEADER
W20260929 16:30:25.834438 21517 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:30:25.834533 21517 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:30:25.834605 21511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:25.835533 21511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:25.836724 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:25.830781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:55431 (local address 127.0.0.1:40293)
0929 16:30:25.830904 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:25.830910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:25.831978 (+ 1068us) server_negotiation.cc:407] Connection header received
0929 16:30:25.832117 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:25.832120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:25.832171 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:25.832261 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:25.832588 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:25.833380 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:25.834517 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:25.834835 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:25.834936 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:25.836082 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:25.836085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:25.836088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:25.836115 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:25.836143 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:25.836156 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:25.836235 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:25.836563 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 16:30:25.836598 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 16:30:25.841952 21511 catalog_manager.cc:1347] Generated new cluster ID: 429a6c89ab004967b6a7d4bbc248d792
I20260929 16:30:25.842031 21511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:25.863873 21511 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:25.865353 21511 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:25.875456 21511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf8e0e63787f49b589a54bd5f6ec4da2: Generated new TSK 0
I20260929 16:30:25.876387 21511 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:25.888150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 21424
W20260929 16:30:25.893882 21409 connection.cc:535] client connection to 127.0.0.1:40293 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40293): Connection reset by peer (error 104)
2026-09-29T16:30:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:25 dist-test-slave-vq9v krb5kdc[21523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:25 dist-test-slave-vq9v krb5kdc[21523](info): set up 2 sockets
Sep 29 16:30:25 dist-test-slave-vq9v krb5kdc[21523](info): commencing operation
krb5kdc: starting...
W20260929 16:30:26.942114 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:27 dist-test-slave-vq9v krb5kdc[21523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:30:27.073247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41153
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:27.193220 21536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:27.193509 21536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:27.193553 21536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:27.200217 21536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:27.200326 21536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:27.200361 21536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:27.200394 21536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:27.200428 21536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:27.207357 21536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:27.208532 21536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:27.209784 21536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:27.218799 21541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:27.218792 21542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:27.218801 21544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:27.219250 21536 server_base.cc:1034] running on GCE node
I20260929 16:30:27.219851 21536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:27.220985 21536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:27.222150 21536 hybrid_clock.cc:648] HybridClock initialized: now 1790699427222138 us; error 68 us; skew 500 ppm
Sep 29 16:30:27 dist-test-slave-vq9v krb5kdc[21523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:27.225075 21536 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:30:27.226174 21536 webserver.cc:460] Webserver started at http://127.0.0.1:42883/ using document root <none> and password file <none>
I20260929 16:30:27.226728 21536 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:27.226791 21536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:27.226961 21536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:27.229354 21536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "c9309d9c0aff47fa98e9f96ab95b6a75"
format_stamp: "Formatted at 2026-09-29 16:30:27 on dist-test-slave-vq9v"
server_key: "c25bc0bf642d920dbeb2bf19275cf9e5"
server_key_iv: "eff2224ec4b82ca3050663eb29d09d2f"
server_key_version: "encryptionkey@0"
I20260929 16:30:27.230036 21536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "c9309d9c0aff47fa98e9f96ab95b6a75"
format_stamp: "Formatted at 2026-09-29 16:30:27 on dist-test-slave-vq9v"
server_key: "c25bc0bf642d920dbeb2bf19275cf9e5"
server_key_iv: "eff2224ec4b82ca3050663eb29d09d2f"
server_key_version: "encryptionkey@0"
I20260929 16:30:27.234156 21536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:30:27.236961 21551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:27.238691 21536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:27.238835 21536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "c9309d9c0aff47fa98e9f96ab95b6a75"
format_stamp: "Formatted at 2026-09-29 16:30:27 on dist-test-slave-vq9v"
server_key: "c25bc0bf642d920dbeb2bf19275cf9e5"
server_key_iv: "eff2224ec4b82ca3050663eb29d09d2f"
server_key_version: "encryptionkey@0"
I20260929 16:30:27.238950 21536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:27.244503 21536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:27.248188 21536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:27.248361 21536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:27.256937 21536 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41153
I20260929 16:30:27.256953 21603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41153 every 8 connection(s)
I20260929 16:30:27.258055 21536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:27.260510 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 21536
I20260929 16:30:27.260622 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:27.261003 392 external_mini_cluster.cc:1428] Setting key e871ea954e07b827949895330d76d3cf
I20260929 16:30:27.261305 21604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:27.267226 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:27.262694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:45307 (local address 127.0.0.1:41153)
0929 16:30:27.263236 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:27.263253 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:27.263290 (+ 37us) server_negotiation.cc:407] Connection header received
0929 16:30:27.263986 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:27.264005 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:27.264338 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:27.264727 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:27.265827 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:27.265833 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:27.265847 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:27.265863 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:27.265926 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:27.265948 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:27.266145 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:27.266230 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 16:30:27.266428 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":166,"threads_started":1}
I20260929 16:30:27.269506 21604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75: Bootstrap starting.
I20260929 16:30:27.270491 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:30:27.272411 21604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:27.273517 21604 log.cc:824] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:27.274916 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:27.272800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:57115 (local address 127.0.0.1:41153)
0929 16:30:27.272946 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:27.272951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:27.273279 (+ 328us) server_negotiation.cc:407] Connection header received
0929 16:30:27.273328 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:27.273330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:27.273365 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:27.273441 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:27.274317 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:27.274320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:27.274324 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:27.274335 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:27.274361 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:27.274375 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:27.274475 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:27.274692 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 16:30:27.274735 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:30:27.276355 21604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75: No bootstrap required, opened a new log
I20260929 16:30:27.280617 21604 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9309d9c0aff47fa98e9f96ab95b6a75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } }
I20260929 16:30:27.280936 21604 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:27.281028 21604 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9309d9c0aff47fa98e9f96ab95b6a75, State: Initialized, Role: FOLLOWER
I20260929 16:30:27.281687 21604 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9309d9c0aff47fa98e9f96ab95b6a75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } }
I20260929 16:30:27.281854 21604 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:27.281931 21604 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:27.282035 21604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:27.283071 21604 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9309d9c0aff47fa98e9f96ab95b6a75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } }
I20260929 16:30:27.283448 21604 leader_election.cc:302] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9309d9c0aff47fa98e9f96ab95b6a75; no voters:
I20260929 16:30:27.283836 21604 leader_election.cc:288] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:27.283983 21614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:27.284296 21614 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75 [term 1 LEADER]: Becoming Leader. State: Replica: c9309d9c0aff47fa98e9f96ab95b6a75, State: Running, Role: LEADER
I20260929 16:30:27.284646 21614 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9309d9c0aff47fa98e9f96ab95b6a75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } }
I20260929 16:30:27.285336 21604 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:27.286842 21616 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9309d9c0aff47fa98e9f96ab95b6a75. Latest consensus state: current_term: 1 leader_uuid: "c9309d9c0aff47fa98e9f96ab95b6a75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9309d9c0aff47fa98e9f96ab95b6a75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } } }
I20260929 16:30:27.286993 21616 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:27.287204 21615 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9309d9c0aff47fa98e9f96ab95b6a75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9309d9c0aff47fa98e9f96ab95b6a75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } } }
I20260929 16:30:27.287312 21615 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:27.287575 21623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:27.290453 21623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:27.296133 21623 catalog_manager.cc:1347] Generated new cluster ID: 6eae1edd8ad54c49a3d615a43b5a8617
I20260929 16:30:27.296227 21623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:27.318118 21623 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:27.318590 21623 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:27.323964 21623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9309d9c0aff47fa98e9f96ab95b6a75: Generated new TSK 0
I20260929 16:30:27.324868 21623 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:27.334676 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 21536
2026-09-29T16:30:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1446 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:27 dist-test-slave-vq9v krb5kdc[21635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:27 dist-test-slave-vq9v krb5kdc[21635](info): set up 2 sockets
Sep 29 16:30:27 dist-test-slave-vq9v krb5kdc[21635](info): commencing operation
krb5kdc: starting...
W20260929 16:30:28.382030 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:28 dist-test-slave-vq9v krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:30:28.508838 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43659
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:28.648984 21648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:28.649367 21648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:28.649451 21648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:28.653707 21648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:28.653784 21648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:28.653805 21648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:28.653821 21648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:28.653837 21648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:28.659883 21648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43659
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:28.661036 21648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:28.662132 21648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:28.671101 21656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:28.671166 21648 server_base.cc:1034] running on GCE node
W20260929 16:30:28.671109 21653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:28.671140 21654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:28.671926 21648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:28.673055 21648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:28.674284 21648 hybrid_clock.cc:648] HybridClock initialized: now 1790699428674264 us; error 30 us; skew 500 ppm
Sep 29 16:30:28 dist-test-slave-vq9v krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:28.677423 21648 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:30:28.678437 21648 webserver.cc:460] Webserver started at http://127.0.0.1:35257/ using document root <none> and password file <none>
I20260929 16:30:28.679034 21648 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:28.679137 21648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:28.679366 21648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:28.681597 21648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "5cb5884a9d3b4adc810bd60230f51707"
format_stamp: "Formatted at 2026-09-29 16:30:28 on dist-test-slave-vq9v"
server_key: "a6411ab8bb6d6b5e714d2af068e43d2d"
server_key_iv: "9cf8e7b583384a671bab7479539f708a"
server_key_version: "encryptionkey@0"
I20260929 16:30:28.682183 21648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "5cb5884a9d3b4adc810bd60230f51707"
format_stamp: "Formatted at 2026-09-29 16:30:28 on dist-test-slave-vq9v"
server_key: "a6411ab8bb6d6b5e714d2af068e43d2d"
server_key_iv: "9cf8e7b583384a671bab7479539f708a"
server_key_version: "encryptionkey@0"
I20260929 16:30:28.686100 21648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:30:28.688743 21663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:28.690176 21648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:28.690333 21648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "5cb5884a9d3b4adc810bd60230f51707"
format_stamp: "Formatted at 2026-09-29 16:30:28 on dist-test-slave-vq9v"
server_key: "a6411ab8bb6d6b5e714d2af068e43d2d"
server_key_iv: "9cf8e7b583384a671bab7479539f708a"
server_key_version: "encryptionkey@0"
I20260929 16:30:28.690479 21648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:28.713034 21648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:28.716516 21648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:28.716740 21648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:28.724346 21648 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43659
I20260929 16:30:28.724366 21715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43659 every 8 connection(s)
I20260929 16:30:28.725437 21648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:28.725975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 21648
I20260929 16:30:28.726121 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:28.726361 392 external_mini_cluster.cc:1428] Setting key 8c6b3092914741745b6700da42ce1707
I20260929 16:30:28.728801 21716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:30:28 dist-test-slave-vq9v krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:30:28.736394 21716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707: Bootstrap starting.
I20260929 16:30:28.739553 21716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:28.740522 21716 log.cc:824] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:28.742797 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:28.727718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:39899 (local address 127.0.0.1:43659)
0929 16:30:28.728151 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:28.728167 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:28.728204 (+ 37us) server_negotiation.cc:407] Connection header received
0929 16:30:28.728844 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:28.728890 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:28.729338 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:28.729642 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:28.730613 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:28.731681 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:28.732962 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:28.733582 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:28.733730 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:28.736925 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:28.736932 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:28.736951 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:28.736989 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:28.739100 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:28.739710 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:28.739718 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:28.739726 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:28.739884 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:28.740282 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:28.740288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:28.740293 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:28.740878 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:28.741072 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:28.741527 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 16:30:28.741838 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":144,"threads_started":1}
I20260929 16:30:28.743086 21716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707: No bootstrap required, opened a new log
I20260929 16:30:28.746424 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:30:28.747403 21716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb5884a9d3b4adc810bd60230f51707" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43659 } }
I20260929 16:30:28.747680 21716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:28.747761 21716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb5884a9d3b4adc810bd60230f51707, State: Initialized, Role: FOLLOWER
I20260929 16:30:28.748207 21716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb5884a9d3b4adc810bd60230f51707" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43659 } }
I20260929 16:30:28.748335 21716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:28.748421 21716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:28.748551 21716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:28.750031 21716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb5884a9d3b4adc810bd60230f51707" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43659 } }
I20260929 16:30:28.750427 21716 leader_election.cc:302] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cb5884a9d3b4adc810bd60230f51707; no voters:
I20260929 16:30:28.750746 21716 leader_election.cc:288] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:28.750852 21726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:28.751089 21726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707 [term 1 LEADER]: Becoming Leader. State: Replica: 5cb5884a9d3b4adc810bd60230f51707, State: Running, Role: LEADER
I20260929 16:30:28.751672 21726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb5884a9d3b4adc810bd60230f51707" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43659 } }
I20260929 16:30:28.752166 21716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:28.753021 21728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cb5884a9d3b4adc810bd60230f51707. Latest consensus state: current_term: 1 leader_uuid: "5cb5884a9d3b4adc810bd60230f51707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb5884a9d3b4adc810bd60230f51707" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43659 } } }
I20260929 16:30:28.753890 21728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:28.753746 21727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cb5884a9d3b4adc810bd60230f51707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb5884a9d3b4adc810bd60230f51707" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43659 } } }
I20260929 16:30:28.754225 21727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707 [sys.catalog]: This master's current role is: LEADER
W20260929 16:30:28.757773 21741 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:30:28.757867 21741 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:30:28.757941 21739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:28.758320 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:28.748641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:38505 (local address 127.0.0.1:43659)
0929 16:30:28.748775 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:28.748779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:28.749038 (+ 259us) server_negotiation.cc:407] Connection header received
0929 16:30:28.749134 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:28.749136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:28.749182 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:28.749327 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:28.750216 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:28.750912 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:28.752060 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:28.752417 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:28.752526 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:28.754496 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:28.754501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:28.754504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:28.754530 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:28.756220 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:28.756808 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:28.756811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:28.756813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:28.756901 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:28.757304 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:28.757309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:28.757312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:28.757538 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:28.757695 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:28.758058 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 16:30:28.758173 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 16:30:28.758898 21739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:28.764834 21739 catalog_manager.cc:1347] Generated new cluster ID: 058c55cf4cdf4d1e8384564f4fd2b74c
I20260929 16:30:28.764925 21739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:28.790954 21739 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:28.792291 21739 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:28.805254 21739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cb5884a9d3b4adc810bd60230f51707: Generated new TSK 0
I20260929 16:30:28.806051 21739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:28.817389 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 21648
W20260929 16:30:28.823156 21633 connection.cc:535] client connection to 127.0.0.1:43659 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43659): Connection reset by peer (error 104)
2026-09-29T16:30:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1482 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:28 dist-test-slave-vq9v krb5kdc[21747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:28 dist-test-slave-vq9v krb5kdc[21747](info): set up 2 sockets
Sep 29 16:30:28 dist-test-slave-vq9v krb5kdc[21747](info): commencing operation
krb5kdc: starting...
W20260929 16:30:29.853576 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:29 dist-test-slave-vq9v krb5kdc[21747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:30:29.984515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46247
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:30.098500 21760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:30.098863 21760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:30.098955 21760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:30.103102 21760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:30.103220 21760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:30.103260 21760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:30.103299 21760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:30.103331 21760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:30.108843 21760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46247
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:30.110029 21760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:30.111083 21760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:30.120316 21765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:30.120438 21760 server_base.cc:1034] running on GCE node
W20260929 16:30:30.120353 21768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:30.120319 21766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:30.121237 21760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:30.122524 21760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:30.123723 21760 hybrid_clock.cc:648] HybridClock initialized: now 1790699430123686 us; error 49 us; skew 500 ppm
Sep 29 16:30:30 dist-test-slave-vq9v krb5kdc[21747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:30.127066 21760 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:30:30.127992 21760 webserver.cc:460] Webserver started at http://127.0.0.1:32771/ using document root <none> and password file <none>
I20260929 16:30:30.128526 21760 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:30.128584 21760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:30.128759 21760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:30.131217 21760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "317542f2009347f1a9658ed08be3ae08"
format_stamp: "Formatted at 2026-09-29 16:30:30 on dist-test-slave-vq9v"
server_key: "349741ad81c6831f49531acf10d987d2"
server_key_iv: "fc082c4b799877e0cd71dbc378c5c7a9"
server_key_version: "encryptionkey@0"
I20260929 16:30:30.131808 21760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "317542f2009347f1a9658ed08be3ae08"
format_stamp: "Formatted at 2026-09-29 16:30:30 on dist-test-slave-vq9v"
server_key: "349741ad81c6831f49531acf10d987d2"
server_key_iv: "fc082c4b799877e0cd71dbc378c5c7a9"
server_key_version: "encryptionkey@0"
I20260929 16:30:30.135876 21760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:30:30.138823 21775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:30.140225 21760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:30:30.140479 21760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "317542f2009347f1a9658ed08be3ae08"
format_stamp: "Formatted at 2026-09-29 16:30:30 on dist-test-slave-vq9v"
server_key: "349741ad81c6831f49531acf10d987d2"
server_key_iv: "fc082c4b799877e0cd71dbc378c5c7a9"
server_key_version: "encryptionkey@0"
I20260929 16:30:30.140667 21760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:30.176725 21760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:30.180330 21760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:30.180568 21760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:30.187827 21760 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46247
I20260929 16:30:30.187897 21827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46247 every 8 connection(s)
I20260929 16:30:30.188860 21760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:30.191803 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 21760
I20260929 16:30:30.191954 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:30.191990 21828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:30.192256 392 external_mini_cluster.cc:1428] Setting key 1ebd6b87abeca935637930e53af3adf8
I20260929 16:30:30.199904 21828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08: Bootstrap starting.
I20260929 16:30:30.202926 21828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:30.202993 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:30.193695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:54503 (local address 127.0.0.1:46247)
0929 16:30:30.194150 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:30.194168 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:30.194214 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:30:30.195054 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:30.195080 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:30.195544 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:30.195952 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:30.196404 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:30.197918 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:30.198909 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:30.199598 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:30.199767 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:30.200822 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:30.200829 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:30.200845 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:30.201415 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:30.201489 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:30.201520 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:30.201664 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:30.201841 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 16:30:30.201980 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":141,"threads_started":1}
I20260929 16:30:30.204021 21828 log.cc:824] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:30.206640 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:30:30.207515 21828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08: No bootstrap required, opened a new log
I20260929 16:30:30.212638 21828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317542f2009347f1a9658ed08be3ae08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46247 } }
I20260929 16:30:30.212992 21828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:30.213083 21828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 317542f2009347f1a9658ed08be3ae08, State: Initialized, Role: FOLLOWER
I20260929 16:30:30.213627 21828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317542f2009347f1a9658ed08be3ae08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46247 } }
I20260929 16:30:30.213814 21828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:30.213912 21828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:30.214032 21828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:30.214623 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:30.209503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:48413 (local address 127.0.0.1:46247)
0929 16:30:30.210403 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:30.210408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:30.210424 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:30:30.210466 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:30.210469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:30.210517 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:30.210611 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:30.210926 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:30.211713 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:30.212624 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:30.212967 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:30.213077 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:30.214167 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:30.214171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:30.214174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:30.214190 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:30.214221 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:30.214236 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:30.214351 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:30.214458 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 16:30:30.214494 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":813}
I20260929 16:30:30.215348 21828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317542f2009347f1a9658ed08be3ae08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46247 } }
I20260929 16:30:30.215755 21828 leader_election.cc:302] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 317542f2009347f1a9658ed08be3ae08; no voters:
I20260929 16:30:30.216079 21828 leader_election.cc:288] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:30.216228 21838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:30.216467 21838 raft_consensus.cc:686] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08 [term 1 LEADER]: Becoming Leader. State: Replica: 317542f2009347f1a9658ed08be3ae08, State: Running, Role: LEADER
I20260929 16:30:30.216789 21838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317542f2009347f1a9658ed08be3ae08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46247 } }
I20260929 16:30:30.217456 21828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:30.218710 21839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "317542f2009347f1a9658ed08be3ae08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317542f2009347f1a9658ed08be3ae08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46247 } } }
I20260929 16:30:30.218844 21839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:30.219270 21847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:30.219228 21840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 317542f2009347f1a9658ed08be3ae08. Latest consensus state: current_term: 1 leader_uuid: "317542f2009347f1a9658ed08be3ae08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317542f2009347f1a9658ed08be3ae08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46247 } } }
I20260929 16:30:30.219377 21840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:30.223361 21847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:30.229025 21847 catalog_manager.cc:1347] Generated new cluster ID: 011a0ffc5c6d49368a5588974a85650a
I20260929 16:30:30.229110 21847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:30.242807 21847 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:30.243917 21847 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:30.255789 21847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 317542f2009347f1a9658ed08be3ae08: Generated new TSK 0
I20260929 16:30:30.256707 21847 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:30.271021 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 21760
W20260929 16:30:30.278321 21745 connection.cc:535] client connection to 127.0.0.1:46247 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46247): Connection reset by peer (error 104)
2026-09-29T16:30:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1455 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:30 dist-test-slave-vq9v krb5kdc[21859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:30 dist-test-slave-vq9v krb5kdc[21859](info): set up 2 sockets
Sep 29 16:30:30 dist-test-slave-vq9v krb5kdc[21859](info): commencing operation
krb5kdc: starting...
W20260929 16:30:31.305763 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:31 dist-test-slave-vq9v krb5kdc[21859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:30:31.442066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39869
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:31.557556 21872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:31.557973 21872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:31.558025 21872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:31.562318 21872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:31.562399 21872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:31.562420 21872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:31.562438 21872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:31.562453 21872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:31.568240 21872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39869
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:31.569458 21872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:31.570518 21872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:31.579671 21878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:31.579770 21872 server_base.cc:1034] running on GCE node
W20260929 16:30:31.579655 21877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:31.579655 21880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:31.580543 21872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:31.581847 21872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:31.583025 21872 hybrid_clock.cc:648] HybridClock initialized: now 1790699431582989 us; error 52 us; skew 500 ppm
Sep 29 16:30:31 dist-test-slave-vq9v krb5kdc[21859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:31.594377 21872 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:30:31.595615 21872 webserver.cc:460] Webserver started at http://127.0.0.1:39387/ using document root <none> and password file <none>
I20260929 16:30:31.596236 21872 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:31.596338 21872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:31.596599 21872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:31.598985 21872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "c0784359436446608906360facedba64"
format_stamp: "Formatted at 2026-09-29 16:30:31 on dist-test-slave-vq9v"
server_key: "c32594d5ec46c54774c3620cb7b62215"
server_key_iv: "8470851154bcd0658c299f9d0d839fbb"
server_key_version: "encryptionkey@0"
I20260929 16:30:31.599641 21872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "c0784359436446608906360facedba64"
format_stamp: "Formatted at 2026-09-29 16:30:31 on dist-test-slave-vq9v"
server_key: "c32594d5ec46c54774c3620cb7b62215"
server_key_iv: "8470851154bcd0658c299f9d0d839fbb"
server_key_version: "encryptionkey@0"
I20260929 16:30:31.603889 21872 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 16:30:31.607008 21887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:31.608760 21872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:30:31.608974 21872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "c0784359436446608906360facedba64"
format_stamp: "Formatted at 2026-09-29 16:30:31 on dist-test-slave-vq9v"
server_key: "c32594d5ec46c54774c3620cb7b62215"
server_key_iv: "8470851154bcd0658c299f9d0d839fbb"
server_key_version: "encryptionkey@0"
I20260929 16:30:31.609155 21872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:31.615289 21872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:31.619442 21872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:31.619696 21872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:31.627338 21872 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39869
I20260929 16:30:31.627348 21939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39869 every 8 connection(s)
I20260929 16:30:31.628341 21872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:31.629223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 21872
I20260929 16:30:31.629369 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:31.629750 392 external_mini_cluster.cc:1428] Setting key e90fbeffc66cef6d5ee948269d9c083f
I20260929 16:30:31.631990 21940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:31.635275 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:31.631237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:56297 (local address 127.0.0.1:39869)
0929 16:30:31.631726 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:31.631742 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:31.631779 (+ 37us) server_negotiation.cc:407] Connection header received
0929 16:30:31.632503 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:31.632522 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:31.632826 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:31.633242 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:31.634061 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:31.634066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:31.634078 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:31.634090 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:31.634149 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:31.634168 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:31.634338 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:31.634411 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 16:30:31.634575 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":142,"threads_started":1}
I20260929 16:30:31.638300 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:30:31.638351 21940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0784359436446608906360facedba64: Bootstrap starting.
I20260929 16:30:31.640921 21940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0784359436446608906360facedba64: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:31.641891 21940 log.cc:824] T 00000000000000000000000000000000 P c0784359436446608906360facedba64: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:31.642554 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:31.640558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:45401 (local address 127.0.0.1:39869)
0929 16:30:31.640707 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:31.640712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:31.640926 (+ 214us) server_negotiation.cc:407] Connection header received
0929 16:30:31.641074 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:31.641077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:31.641120 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:31.641240 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:31.642080 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:31.642084 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:31.642087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:31.642106 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:31.642153 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:31.642164 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:31.642267 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:31.642376 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 16:30:31.642422 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:30:31.644490 21940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0784359436446608906360facedba64: No bootstrap required, opened a new log
I20260929 16:30:31.648558 21940 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0784359436446608906360facedba64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0784359436446608906360facedba64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39869 } }
I20260929 16:30:31.648885 21940 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0784359436446608906360facedba64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:31.648942 21940 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0784359436446608906360facedba64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0784359436446608906360facedba64, State: Initialized, Role: FOLLOWER
I20260929 16:30:31.649430 21940 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0784359436446608906360facedba64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0784359436446608906360facedba64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39869 } }
I20260929 16:30:31.649566 21940 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0784359436446608906360facedba64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:31.649616 21940 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0784359436446608906360facedba64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:31.649688 21940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0784359436446608906360facedba64 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:31.650658 21940 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0784359436446608906360facedba64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0784359436446608906360facedba64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39869 } }
I20260929 16:30:31.650974 21940 leader_election.cc:302] T 00000000000000000000000000000000 P c0784359436446608906360facedba64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0784359436446608906360facedba64; no voters:
I20260929 16:30:31.651304 21940 leader_election.cc:288] T 00000000000000000000000000000000 P c0784359436446608906360facedba64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:31.651446 21950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0784359436446608906360facedba64 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:31.651706 21950 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0784359436446608906360facedba64 [term 1 LEADER]: Becoming Leader. State: Replica: c0784359436446608906360facedba64, State: Running, Role: LEADER
I20260929 16:30:31.652091 21950 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0784359436446608906360facedba64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0784359436446608906360facedba64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39869 } }
I20260929 16:30:31.652642 21940 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0784359436446608906360facedba64 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:31.655164 21951 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0784359436446608906360facedba64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0784359436446608906360facedba64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0784359436446608906360facedba64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39869 } } }
I20260929 16:30:31.655304 21951 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0784359436446608906360facedba64 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:31.655642 21952 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0784359436446608906360facedba64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0784359436446608906360facedba64. Latest consensus state: current_term: 1 leader_uuid: "c0784359436446608906360facedba64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0784359436446608906360facedba64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39869 } } }
I20260929 16:30:31.655810 21952 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0784359436446608906360facedba64 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:31.656149 21960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:31.659075 21960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:31.665026 21960 catalog_manager.cc:1347] Generated new cluster ID: 5ad57357cfd4476c876df8155c035eb3
I20260929 16:30:31.665123 21960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:31.679275 21960 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:31.679661 21960 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:31.685709 21960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0784359436446608906360facedba64: Generated new TSK 0
I20260929 16:30:31.686585 21960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:31.697777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 21872
2026-09-29T16:30:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:31 dist-test-slave-vq9v krb5kdc[21971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:31 dist-test-slave-vq9v krb5kdc[21971](info): set up 2 sockets
Sep 29 16:30:31 dist-test-slave-vq9v krb5kdc[21971](info): commencing operation
krb5kdc: starting...
W20260929 16:30:32.749647 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:32 dist-test-slave-vq9v krb5kdc[21971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:30:32.877416 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42311
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:32.989357 21984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:32.989670 21984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:32.989786 21984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:32.994365 21984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:32.994431 21984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:32.994451 21984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:32.994468 21984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:32.994518 21984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:32.999948 21984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42311
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:33.001034 21984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:33.002111 21984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:33.011009 21990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:33.011019 21989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:33.011022 21992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:33.011512 21984 server_base.cc:1034] running on GCE node
I20260929 16:30:33.012086 21984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:33.013360 21984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:33.014569 21984 hybrid_clock.cc:648] HybridClock initialized: now 1790699433014543 us; error 38 us; skew 500 ppm
Sep 29 16:30:33 dist-test-slave-vq9v krb5kdc[21971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:33.017691 21984 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:30:33.018751 21984 webserver.cc:460] Webserver started at http://127.0.98.62:37811/ using document root <none> and password file <none>
I20260929 16:30:33.019381 21984 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:33.019449 21984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:33.019682 21984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:33.022037 21984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "cf9b28021b6c4a33a1b7e5ad858440d1"
format_stamp: "Formatted at 2026-09-29 16:30:33 on dist-test-slave-vq9v"
server_key: "8e7adc6c28cad0446883cc793c1a51a4"
server_key_iv: "2fb5657938ec1edee8e3c87d7a7c7dce"
server_key_version: "encryptionkey@0"
I20260929 16:30:33.022670 21984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "cf9b28021b6c4a33a1b7e5ad858440d1"
format_stamp: "Formatted at 2026-09-29 16:30:33 on dist-test-slave-vq9v"
server_key: "8e7adc6c28cad0446883cc793c1a51a4"
server_key_iv: "2fb5657938ec1edee8e3c87d7a7c7dce"
server_key_version: "encryptionkey@0"
I20260929 16:30:33.026876 21984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:30:33.029733 21999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:33.031111 21984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:33.031278 21984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "cf9b28021b6c4a33a1b7e5ad858440d1"
format_stamp: "Formatted at 2026-09-29 16:30:33 on dist-test-slave-vq9v"
server_key: "8e7adc6c28cad0446883cc793c1a51a4"
server_key_iv: "2fb5657938ec1edee8e3c87d7a7c7dce"
server_key_version: "encryptionkey@0"
I20260929 16:30:33.031400 21984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:33.056268 21984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:33.059743 21984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:33.059985 21984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:33.067240 21984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42311
I20260929 16:30:33.067265 22051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42311 every 8 connection(s)
I20260929 16:30:33.068255 21984 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:33.071098 22052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:33.074306 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 21984
I20260929 16:30:33.074447 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:33.074752 392 external_mini_cluster.cc:1428] Setting key a450f64602e0fa6e42a9e65316307b8e
I20260929 16:30:33.077600 22052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1: Bootstrap starting.
I20260929 16:30:33.079891 22052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:33.080669 22052 log.cc:824] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:33.082849 22052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1: No bootstrap required, opened a new log
Sep 29 16:30:33 dist-test-slave-vq9v krb5kdc[21971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:30:33.087119 22052 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9b28021b6c4a33a1b7e5ad858440d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42311 } }
I20260929 16:30:33.087387 22052 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:33.087466 22052 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf9b28021b6c4a33a1b7e5ad858440d1, State: Initialized, Role: FOLLOWER
I20260929 16:30:33.088012 22052 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9b28021b6c4a33a1b7e5ad858440d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42311 } }
I20260929 16:30:33.088145 22052 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:33.088240 22052 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:33.088357 22052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:33.089475 22052 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9b28021b6c4a33a1b7e5ad858440d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42311 } }
I20260929 16:30:33.089847 22052 leader_election.cc:302] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf9b28021b6c4a33a1b7e5ad858440d1; no voters:
I20260929 16:30:33.090167 22052 leader_election.cc:288] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:33.090314 22057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:33.090468 22055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:33.076306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46979 (local address 127.0.98.62:42311)
0929 16:30:33.076773 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:33.076787 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:33.076823 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:30:33.077530 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:33.077549 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:33.077958 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:33.078319 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:33.079144 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:33.080210 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:33.081129 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:33.081680 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:33.081855 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:33.084830 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:33.084835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:33.084850 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:33.084881 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:33.086732 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:33.087339 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:33.087345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:33.087350 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:33.087569 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:33.087942 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:33.087945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:33.087947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:33.088551 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:33.088804 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:33.089155 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 16:30:33.089444 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":134,"threads_started":1}
I20260929 16:30:33.090541 22057 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1 [term 1 LEADER]: Becoming Leader. State: Replica: cf9b28021b6c4a33a1b7e5ad858440d1, State: Running, Role: LEADER
I20260929 16:30:33.090864 22057 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9b28021b6c4a33a1b7e5ad858440d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42311 } }
I20260929 16:30:33.091486 22052 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:33.092222 22057 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf9b28021b6c4a33a1b7e5ad858440d1. Latest consensus state: current_term: 1 leader_uuid: "cf9b28021b6c4a33a1b7e5ad858440d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9b28021b6c4a33a1b7e5ad858440d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42311 } } }
I20260929 16:30:33.092403 22057 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:33.092672 22059 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf9b28021b6c4a33a1b7e5ad858440d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9b28021b6c4a33a1b7e5ad858440d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42311 } } }
I20260929 16:30:33.092789 22059 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:33.092839 22065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:33.093971 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:30:33.096208 22065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:33.103289 22065 catalog_manager.cc:1347] Generated new cluster ID: 6411d2e962124d14a8d6350044aaf2bf
I20260929 16:30:33.103466 22065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:33.109881 22055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:33.097703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39911 (local address 127.0.98.62:42311)
0929 16:30:33.097837 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:33.097842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:33.100459 (+ 2617us) server_negotiation.cc:407] Connection header received
0929 16:30:33.100658 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:33.100661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:33.100717 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:33.100807 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:33.101836 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:33.102716 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:33.103795 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:33.104187 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:33.104322 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:33.106056 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:33.106061 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:33.106064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:33.106094 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:33.107917 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:33.108493 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:33.108497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:33.108499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:33.108592 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:33.108915 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:33.108918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:33.108919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:33.109140 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:33.109324 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:33.109588 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 16:30:33.109701 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":51}
I20260929 16:30:33.113765 22065 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:33.114988 22065 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:33.127583 22065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf9b28021b6c4a33a1b7e5ad858440d1: Generated new TSK 0
I20260929 16:30:33.128413 22065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:33.135630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 21984
W20260929 16:30:33.141270 21969 connection.cc:535] client connection to 127.0.98.62:42311 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:42311): Connection reset by peer (error 104)
2026-09-29T16:30:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:33 dist-test-slave-vq9v krb5kdc[22083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:33 dist-test-slave-vq9v krb5kdc[22083](info): set up 2 sockets
Sep 29 16:30:33 dist-test-slave-vq9v krb5kdc[22083](info): commencing operation
krb5kdc: starting...
W20260929 16:30:34.189607 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:34 dist-test-slave-vq9v krb5kdc[22083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:30:34.325979 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45993
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:34.443038 22096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:34.443396 22096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:34.443488 22096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:34.447700 22096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:34.447824 22096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:34.447866 22096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:34.447904 22096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:34.447937 22096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:34.453789 22096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45993
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:34.455061 22096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:34.456212 22096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:34.465456 22101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:34.465456 22102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:34.465476 22104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:34.465911 22096 server_base.cc:1034] running on GCE node
I20260929 16:30:34.466544 22096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:34.467741 22096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:34.468961 22096 hybrid_clock.cc:648] HybridClock initialized: now 1790699434468943 us; error 32 us; skew 500 ppm
Sep 29 16:30:34 dist-test-slave-vq9v krb5kdc[22083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:34.471995 22096 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:30:34.473040 22096 webserver.cc:460] Webserver started at http://127.0.98.62:39367/ using document root <none> and password file <none>
I20260929 16:30:34.473655 22096 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:34.473721 22096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:34.473892 22096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:34.475970 22096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "e851e1316d1d4e3ca2aacda07fe54f12"
format_stamp: "Formatted at 2026-09-29 16:30:34 on dist-test-slave-vq9v"
server_key: "6848956eaaa16783a3164e34c04d4d8a"
server_key_iv: "1860cc35b152d503c7fa101c4100b8d0"
server_key_version: "encryptionkey@0"
I20260929 16:30:34.476500 22096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "e851e1316d1d4e3ca2aacda07fe54f12"
format_stamp: "Formatted at 2026-09-29 16:30:34 on dist-test-slave-vq9v"
server_key: "6848956eaaa16783a3164e34c04d4d8a"
server_key_iv: "1860cc35b152d503c7fa101c4100b8d0"
server_key_version: "encryptionkey@0"
I20260929 16:30:34.480419 22096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:30:34.483083 22111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:34.484532 22096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:30:34.484714 22096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "e851e1316d1d4e3ca2aacda07fe54f12"
format_stamp: "Formatted at 2026-09-29 16:30:34 on dist-test-slave-vq9v"
server_key: "6848956eaaa16783a3164e34c04d4d8a"
server_key_iv: "1860cc35b152d503c7fa101c4100b8d0"
server_key_version: "encryptionkey@0"
I20260929 16:30:34.484851 22096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:34.520155 22096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:34.524147 22096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:34.524312 22096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:34.532351 22096 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45993
I20260929 16:30:34.532390 22163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45993 every 8 connection(s)
I20260929 16:30:34.533479 22096 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:34.533860 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 22096
I20260929 16:30:34.534032 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:34.534349 392 external_mini_cluster.cc:1428] Setting key 4262bf44808b4da9893c641eea6767a0
I20260929 16:30:34.536849 22164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:34.542721 22164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12: Bootstrap starting.
I20260929 16:30:34.543807 22167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:34.535863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56197 (local address 127.0.98.62:45993)
0929 16:30:34.536352 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:34.536368 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:34.536406 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:30:34.537159 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:34.537179 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:34.537602 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:34.537967 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:34.538396 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:34.539409 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:34.540334 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:34.540828 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:34.541003 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:34.542015 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:34.542020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:34.542034 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:34.542452 (+ 418us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:34.542512 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:34.542535 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:34.542720 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:34.542888 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 16:30:34.543040 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":149,"threads_started":1}
I20260929 16:30:34.545287 22164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:34.546149 22164 log.cc:824] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:34.547055 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:30:34.548313 22164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12: No bootstrap required, opened a new log
I20260929 16:30:34.553228 22164 raft_consensus.cc:348] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e851e1316d1d4e3ca2aacda07fe54f12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45993 } }
I20260929 16:30:34.553565 22167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:34.549134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52965 (local address 127.0.98.62:45993)
0929 16:30:34.549291 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:34.549295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:34.549474 (+ 179us) server_negotiation.cc:407] Connection header received
0929 16:30:34.549593 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:34.549595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:34.549635 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:34.549714 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:34.550010 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:34.550528 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:34.551715 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:34.551943 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:34.552018 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:34.552950 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:34.552953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:34.552955 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:34.552969 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:34.552995 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:34.553007 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:34.553138 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:34.553336 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 16:30:34.553400 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:30:34.553586 22164 raft_consensus.cc:374] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:34.553704 22164 raft_consensus.cc:729] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e851e1316d1d4e3ca2aacda07fe54f12, State: Initialized, Role: FOLLOWER
I20260929 16:30:34.554139 22164 consensus_queue.cc:260] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e851e1316d1d4e3ca2aacda07fe54f12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45993 } }
I20260929 16:30:34.554332 22164 raft_consensus.cc:388] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:34.554425 22164 raft_consensus.cc:482] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:34.554531 22164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:34.555543 22164 raft_consensus.cc:504] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e851e1316d1d4e3ca2aacda07fe54f12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45993 } }
I20260929 16:30:34.555871 22164 leader_election.cc:302] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e851e1316d1d4e3ca2aacda07fe54f12; no voters:
I20260929 16:30:34.556147 22164 leader_election.cc:288] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:34.556320 22174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:34.556550 22174 raft_consensus.cc:686] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12 [term 1 LEADER]: Becoming Leader. State: Replica: e851e1316d1d4e3ca2aacda07fe54f12, State: Running, Role: LEADER
I20260929 16:30:34.556948 22174 consensus_queue.cc:237] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e851e1316d1d4e3ca2aacda07fe54f12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45993 } }
I20260929 16:30:34.557545 22164 sys_catalog.cc:564] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:34.559074 22176 sys_catalog.cc:455] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e851e1316d1d4e3ca2aacda07fe54f12. Latest consensus state: current_term: 1 leader_uuid: "e851e1316d1d4e3ca2aacda07fe54f12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e851e1316d1d4e3ca2aacda07fe54f12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45993 } } }
I20260929 16:30:34.559211 22176 sys_catalog.cc:458] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:34.559571 22186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:34.560249 22175 sys_catalog.cc:455] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e851e1316d1d4e3ca2aacda07fe54f12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e851e1316d1d4e3ca2aacda07fe54f12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45993 } } }
I20260929 16:30:34.560425 22175 sys_catalog.cc:458] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:34.562507 22186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:34.568413 22186 catalog_manager.cc:1347] Generated new cluster ID: fade3a5043c44327933cfcc334db2424
I20260929 16:30:34.568498 22186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:34.575544 22186 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:34.576823 22186 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:34.584141 22186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e851e1316d1d4e3ca2aacda07fe54f12: Generated new TSK 0
I20260929 16:30:34.585014 22186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:34.589831 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 22096
W20260929 16:30:34.595219 22081 connection.cc:535] client connection to 127.0.98.62:45993 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:45993): Connection reset by peer (error 104)
2026-09-29T16:30:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1454 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:34 dist-test-slave-vq9v krb5kdc[22195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:34 dist-test-slave-vq9v krb5kdc[22195](info): set up 2 sockets
Sep 29 16:30:34 dist-test-slave-vq9v krb5kdc[22195](info): commencing operation
krb5kdc: starting...
W20260929 16:30:35.645545 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:35 dist-test-slave-vq9v krb5kdc[22195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:30:35.776527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37579
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:35.894522 22208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:35.894834 22208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:35.894877 22208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:35.899068 22208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:35.899148 22208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:35.899168 22208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:35.899185 22208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:35.899201 22208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:35.904877 22208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37579
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:35.906097 22208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:35.907182 22208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:30:35.916721 22208 server_base.cc:1034] running on GCE node
W20260929 16:30:35.916661 22213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:35.916640 22216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:35.916636 22214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:35.917614 22208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:35.918910 22208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:35.920133 22208 hybrid_clock.cc:648] HybridClock initialized: now 1790699435920112 us; error 41 us; skew 500 ppm
Sep 29 16:30:35 dist-test-slave-vq9v krb5kdc[22195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:35.923832 22208 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:30:35.924894 22208 webserver.cc:460] Webserver started at http://127.0.98.62:36577/ using document root <none> and password file <none>
I20260929 16:30:35.925566 22208 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:35.925639 22208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:35.925827 22208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:35.928090 22208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "eb6fb17a363a433c8d0c5b2fb01933cb"
format_stamp: "Formatted at 2026-09-29 16:30:35 on dist-test-slave-vq9v"
server_key: "cbb485f25e4fa6c820092209ef10bff9"
server_key_iv: "5dd8cfa1baa1f8636297d26b442e4098"
server_key_version: "encryptionkey@0"
I20260929 16:30:35.928683 22208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "eb6fb17a363a433c8d0c5b2fb01933cb"
format_stamp: "Formatted at 2026-09-29 16:30:35 on dist-test-slave-vq9v"
server_key: "cbb485f25e4fa6c820092209ef10bff9"
server_key_iv: "5dd8cfa1baa1f8636297d26b442e4098"
server_key_version: "encryptionkey@0"
I20260929 16:30:35.932924 22208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:30:35.935791 22223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:35.937326 22208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:35.937470 22208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "eb6fb17a363a433c8d0c5b2fb01933cb"
format_stamp: "Formatted at 2026-09-29 16:30:35 on dist-test-slave-vq9v"
server_key: "cbb485f25e4fa6c820092209ef10bff9"
server_key_iv: "5dd8cfa1baa1f8636297d26b442e4098"
server_key_version: "encryptionkey@0"
I20260929 16:30:35.937603 22208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:35.942746 22208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:35.946267 22208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:35.946477 22208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:35.954182 22208 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37579
I20260929 16:30:35.954200 22275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37579 every 8 connection(s)
I20260929 16:30:35.955240 22208 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:35.958422 22276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:35.963690 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 22208
I20260929 16:30:35.963838 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:35.964151 392 external_mini_cluster.cc:1428] Setting key e19eafd874658ce20a230823c53a95d3
I20260929 16:30:35.964512 22276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb: Bootstrap starting.
I20260929 16:30:35.967140 22276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:35.967935 22276 log.cc:824] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:35.969741 22279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:35.965685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49707 (local address 127.0.98.62:37579)
0929 16:30:35.966205 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:35.966220 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:35.966257 (+ 37us) server_negotiation.cc:407] Connection header received
0929 16:30:35.966893 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:35.966916 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:35.967214 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:35.967621 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:35.968466 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:35.968472 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:35.968493 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:35.968504 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:35.968558 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:35.968577 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:35.968693 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:35.968792 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 16:30:35.968950 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":139,"threads_started":1}
I20260929 16:30:35.970171 22276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb: No bootstrap required, opened a new log
I20260929 16:30:35.972831 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:30:35.974502 22276 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6fb17a363a433c8d0c5b2fb01933cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37579 } }
I20260929 16:30:35.974773 22276 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:35.974854 22276 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6fb17a363a433c8d0c5b2fb01933cb, State: Initialized, Role: FOLLOWER
I20260929 16:30:35.975425 22276 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6fb17a363a433c8d0c5b2fb01933cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37579 } }
I20260929 16:30:35.975615 22276 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:35.975687 22276 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:35.975787 22276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:35.976675 22279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:35.974991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47835 (local address 127.0.98.62:37579)
0929 16:30:35.975176 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:35.975181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:35.975249 (+ 68us) server_negotiation.cc:407] Connection header received
0929 16:30:35.975356 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:35.975359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:35.975397 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:35.975486 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:35.976223 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:35.976226 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:35.976229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:35.976240 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:35.976267 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:35.976279 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:35.976372 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:35.976525 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 16:30:35.976564 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 16:30:35.977094 22276 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6fb17a363a433c8d0c5b2fb01933cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37579 } }
I20260929 16:30:35.977488 22276 leader_election.cc:302] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb6fb17a363a433c8d0c5b2fb01933cb; no voters:
I20260929 16:30:35.977895 22276 leader_election.cc:288] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:35.977960 22286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:35.978169 22286 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb [term 1 LEADER]: Becoming Leader. State: Replica: eb6fb17a363a433c8d0c5b2fb01933cb, State: Running, Role: LEADER
I20260929 16:30:35.978479 22286 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6fb17a363a433c8d0c5b2fb01933cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37579 } }
I20260929 16:30:35.979249 22276 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:35.980248 22287 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb6fb17a363a433c8d0c5b2fb01933cb. Latest consensus state: current_term: 1 leader_uuid: "eb6fb17a363a433c8d0c5b2fb01933cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6fb17a363a433c8d0c5b2fb01933cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37579 } } }
I20260929 16:30:35.980374 22287 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:35.980692 22288 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb6fb17a363a433c8d0c5b2fb01933cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6fb17a363a433c8d0c5b2fb01933cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37579 } } }
I20260929 16:30:35.980777 22288 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:35.981184 22295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:35.984403 22295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:35.990151 22295 catalog_manager.cc:1347] Generated new cluster ID: 045f40b7446c487c84768dea1360ab63
I20260929 16:30:35.990250 22295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:36.006589 22295 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:36.006999 22295 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:36.015498 22295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb6fb17a363a433c8d0c5b2fb01933cb: Generated new TSK 0
I20260929 16:30:36.016426 22295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:36.031158 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 22208
2026-09-29T16:30:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1442 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:36 dist-test-slave-vq9v krb5kdc[22307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:36 dist-test-slave-vq9v krb5kdc[22307](info): set up 2 sockets
Sep 29 16:30:36 dist-test-slave-vq9v krb5kdc[22307](info): commencing operation
krb5kdc: starting...
W20260929 16:30:37.073688 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:37 dist-test-slave-vq9v krb5kdc[22307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:30:37.205098 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43881
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:37.324064 22320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:37.324378 22320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:37.324424 22320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:37.328629 22320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:37.328725 22320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:37.328747 22320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:37.328765 22320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:37.328781 22320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:37.334522 22320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:37.335716 22320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:37.336797 22320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:30:37.345674 22320 server_base.cc:1034] running on GCE node
W20260929 16:30:37.345639 22326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:37.345732 22325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:37.345633 22328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:37.346590 22320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:37.347883 22320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:37.349103 22320 hybrid_clock.cc:648] HybridClock initialized: now 1790699437349084 us; error 35 us; skew 500 ppm
Sep 29 16:30:37 dist-test-slave-vq9v krb5kdc[22307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:37.362226 22320 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:30:37.363405 22320 webserver.cc:460] Webserver started at http://127.0.98.62:44269/ using document root <none> and password file <none>
I20260929 16:30:37.364039 22320 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:37.364157 22320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:37.364427 22320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:37.366747 22320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "41964a298afc49668e3a1d64f33bfb07"
format_stamp: "Formatted at 2026-09-29 16:30:37 on dist-test-slave-vq9v"
server_key: "84ab89b6f43ca3e3c502c9b4c9b2eab1"
server_key_iv: "e2d096b0dfc6abd83a405dba7443bb58"
server_key_version: "encryptionkey@0"
I20260929 16:30:37.367462 22320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "41964a298afc49668e3a1d64f33bfb07"
format_stamp: "Formatted at 2026-09-29 16:30:37 on dist-test-slave-vq9v"
server_key: "84ab89b6f43ca3e3c502c9b4c9b2eab1"
server_key_iv: "e2d096b0dfc6abd83a405dba7443bb58"
server_key_version: "encryptionkey@0"
I20260929 16:30:37.371667 22320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:30:37.374378 22335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:37.375705 22320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:37.375882 22320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "41964a298afc49668e3a1d64f33bfb07"
format_stamp: "Formatted at 2026-09-29 16:30:37 on dist-test-slave-vq9v"
server_key: "84ab89b6f43ca3e3c502c9b4c9b2eab1"
server_key_iv: "e2d096b0dfc6abd83a405dba7443bb58"
server_key_version: "encryptionkey@0"
I20260929 16:30:37.376026 22320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:37.387601 22320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:37.391417 22320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:37.391774 22320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:37.400303 22320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43881
I20260929 16:30:37.400326 22387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43881 every 8 connection(s)
I20260929 16:30:37.401546 22320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:37.402240 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 22320
I20260929 16:30:37.402395 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:37.402738 392 external_mini_cluster.cc:1428] Setting key ae81a39cde1689c9ef28e39ee398c09b
I20260929 16:30:37.404959 22388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:37.412708 22388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07: Bootstrap starting.
Sep 29 16:30:37 dist-test-slave-vq9v krb5kdc[22307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:30:37.415453 22388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:37.416286 22388 log.cc:824] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:37.418712 22388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07: No bootstrap required, opened a new log
I20260929 16:30:37.419466 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:37.404361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48089 (local address 127.0.98.62:43881)
0929 16:30:37.404834 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:37.404851 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:37.404888 (+ 37us) server_negotiation.cc:407] Connection header received
0929 16:30:37.405667 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:37.405694 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:37.406074 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:37.406388 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:37.407494 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:37.408597 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:37.409910 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:37.410566 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:37.410767 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:37.414163 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:37.414169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:37.414186 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:37.414222 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:37.416253 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:37.416806 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:37.416811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:37.416817 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:37.416947 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:37.417316 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:37.417320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:37.417323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:37.417772 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:37.417996 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:37.418306 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 16:30:37.418630 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":163,"threads_started":1}
I20260929 16:30:37.423117 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:30:37.423378 22388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41964a298afc49668e3a1d64f33bfb07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43881 } }
I20260929 16:30:37.423679 22388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:37.423784 22388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41964a298afc49668e3a1d64f33bfb07, State: Initialized, Role: FOLLOWER
I20260929 16:30:37.424263 22388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41964a298afc49668e3a1d64f33bfb07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43881 } }
I20260929 16:30:37.424432 22388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:37.424572 22388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:37.424754 22388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:37.426239 22388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41964a298afc49668e3a1d64f33bfb07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43881 } }
I20260929 16:30:37.426687 22388 leader_election.cc:302] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41964a298afc49668e3a1d64f33bfb07; no voters:
I20260929 16:30:37.427114 22388 leader_election.cc:288] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:37.427265 22398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:37.427547 22398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07 [term 1 LEADER]: Becoming Leader. State: Replica: 41964a298afc49668e3a1d64f33bfb07, State: Running, Role: LEADER
I20260929 16:30:37.427933 22398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41964a298afc49668e3a1d64f33bfb07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43881 } }
I20260929 16:30:37.428612 22388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:37.429345 22398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41964a298afc49668e3a1d64f33bfb07. Latest consensus state: current_term: 1 leader_uuid: "41964a298afc49668e3a1d64f33bfb07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41964a298afc49668e3a1d64f33bfb07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43881 } } }
I20260929 16:30:37.430143 22398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:37.430025 22399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41964a298afc49668e3a1d64f33bfb07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41964a298afc49668e3a1d64f33bfb07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43881 } } }
I20260929 16:30:37.430496 22399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07 [sys.catalog]: This master's current role is: LEADER
W20260929 16:30:37.434979 22413 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:30:37.435075 22413 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:30:37.435160 22407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:37.435887 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:37.425587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36741 (local address 127.0.98.62:43881)
0929 16:30:37.425732 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:37.425737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:37.425911 (+ 174us) server_negotiation.cc:407] Connection header received
0929 16:30:37.426006 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:37.426012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:37.426080 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:37.426196 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:37.427084 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:37.427871 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:37.429078 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:37.429539 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:37.429688 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:37.431849 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:37.431854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:37.431857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:37.431888 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:37.433564 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:37.434083 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:37.434087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:37.434089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:37.434180 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:37.434488 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:37.434491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:37.434493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:37.434736 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:37.434990 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:37.435656 (+ 666us) server_negotiation.cc:299] Negotiation successful
0929 16:30:37.435769 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:30:37.436079 22407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:37.442116 22407 catalog_manager.cc:1347] Generated new cluster ID: 81060a11d23d4303b0d01462b14d463c
I20260929 16:30:37.442206 22407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:37.449466 22407 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:37.450438 22407 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:37.454290 22407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41964a298afc49668e3a1d64f33bfb07: Generated new TSK 0
I20260929 16:30:37.455083 22407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:37.464692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 22320
2026-09-29T16:30:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:37 dist-test-slave-vq9v krb5kdc[22419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:37 dist-test-slave-vq9v krb5kdc[22419](info): set up 2 sockets
Sep 29 16:30:37 dist-test-slave-vq9v krb5kdc[22419](info): commencing operation
krb5kdc: starting...
W20260929 16:30:38.513586 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:38 dist-test-slave-vq9v krb5kdc[22419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:30:38.650439 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45273
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:38.781257 22432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:38.781563 22432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:38.781607 22432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:38.785622 22432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:38.785689 22432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:38.785708 22432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:38.785730 22432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:38.785746 22432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:38.791234 22432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:38.792284 22432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:38.793272 22432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:38.801945 22437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:38.802055 22432 server_base.cc:1034] running on GCE node
W20260929 16:30:38.801950 22438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:38.802033 22440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:38.802950 22432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:38.804137 22432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:38.805305 22432 hybrid_clock.cc:648] HybridClock initialized: now 1790699438805270 us; error 49 us; skew 500 ppm
Sep 29 16:30:38 dist-test-slave-vq9v krb5kdc[22419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:38.808192 22432 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:30:38.809151 22432 webserver.cc:460] Webserver started at http://127.0.98.62:43297/ using document root <none> and password file <none>
I20260929 16:30:38.809789 22432 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:38.809890 22432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:38.810137 22432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:38.812276 22432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "0ba6a28de51240cbacc0597226d6a9a9"
format_stamp: "Formatted at 2026-09-29 16:30:38 on dist-test-slave-vq9v"
server_key: "c3403573151ebb14d474e2475ef485b0"
server_key_iv: "e759a3bd38401809a3c9facad2303b6f"
server_key_version: "encryptionkey@0"
I20260929 16:30:38.812866 22432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "0ba6a28de51240cbacc0597226d6a9a9"
format_stamp: "Formatted at 2026-09-29 16:30:38 on dist-test-slave-vq9v"
server_key: "c3403573151ebb14d474e2475ef485b0"
server_key_iv: "e759a3bd38401809a3c9facad2303b6f"
server_key_version: "encryptionkey@0"
I20260929 16:30:38.817298 22432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260929 16:30:38.820043 22447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:38.821408 22432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:38.821549 22432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "0ba6a28de51240cbacc0597226d6a9a9"
format_stamp: "Formatted at 2026-09-29 16:30:38 on dist-test-slave-vq9v"
server_key: "c3403573151ebb14d474e2475ef485b0"
server_key_iv: "e759a3bd38401809a3c9facad2303b6f"
server_key_version: "encryptionkey@0"
I20260929 16:30:38.821688 22432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:38.847589 22432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:38.851619 22432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:38.851872 22432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:38.859946 22432 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45273
I20260929 16:30:38.859990 22499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45273 every 8 connection(s)
I20260929 16:30:38.860992 22432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:38.864159 22500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:38.868597 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 22432
I20260929 16:30:38.868780 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:38.869149 392 external_mini_cluster.cc:1428] Setting key e96a1f593f34913efe5ec86d74deaf9a
I20260929 16:30:38.871179 22500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9: Bootstrap starting.
I20260929 16:30:38.873771 22500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:38.874631 22500 log.cc:824] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:38.876820 22500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9: No bootstrap required, opened a new log
I20260929 16:30:38.879271 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:38.870742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47237 (local address 127.0.98.62:45273)
0929 16:30:38.871227 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:38.871256 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:38.871481 (+ 225us) server_negotiation.cc:407] Connection header received
0929 16:30:38.872309 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:38.872327 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:38.872698 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:38.873070 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:38.873547 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:38.874663 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:38.875620 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:38.876139 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:38.876335 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:38.877266 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:38.877272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:38.877297 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:38.877705 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:38.877764 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:38.877789 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:38.877903 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:38.878027 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 16:30:38.878264 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":152,"threads_started":1}
I20260929 16:30:38.880704 22500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba6a28de51240cbacc0597226d6a9a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45273 } }
I20260929 16:30:38.880985 22500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:38.881081 22500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba6a28de51240cbacc0597226d6a9a9, State: Initialized, Role: FOLLOWER
I20260929 16:30:38.881692 22500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba6a28de51240cbacc0597226d6a9a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45273 } }
I20260929 16:30:38.881896 22500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:38.881976 22500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:38.882086 22500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:38.882638 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:30:38.883252 22500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba6a28de51240cbacc0597226d6a9a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45273 } }
I20260929 16:30:38.883651 22500 leader_election.cc:302] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ba6a28de51240cbacc0597226d6a9a9; no voters:
I20260929 16:30:38.884057 22500 leader_election.cc:288] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:38.884138 22507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:38.884369 22507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9 [term 1 LEADER]: Becoming Leader. State: Replica: 0ba6a28de51240cbacc0597226d6a9a9, State: Running, Role: LEADER
I20260929 16:30:38.885468 22507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba6a28de51240cbacc0597226d6a9a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45273 } }
I20260929 16:30:38.885684 22500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:38.887250 22511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ba6a28de51240cbacc0597226d6a9a9. Latest consensus state: current_term: 1 leader_uuid: "0ba6a28de51240cbacc0597226d6a9a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba6a28de51240cbacc0597226d6a9a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45273 } } }
I20260929 16:30:38.887392 22511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:38.887835 22521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:38.887844 22510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ba6a28de51240cbacc0597226d6a9a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba6a28de51240cbacc0597226d6a9a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45273 } } }
I20260929 16:30:38.889326 22510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:38.890169 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:38.885333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35087 (local address 127.0.98.62:45273)
0929 16:30:38.885476 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:38.885479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:38.885490 (+ 11us) server_negotiation.cc:407] Connection header received
0929 16:30:38.885522 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:38.885525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:38.885563 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:38.885680 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:38.886000 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:38.886574 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:38.887817 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:38.888162 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:38.888260 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:38.889392 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:38.889395 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:38.889398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:38.889413 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:38.889442 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:38.889455 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:38.889518 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:38.889953 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 16:30:38.890015 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:30:38.891650 22521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:38.897639 22521 catalog_manager.cc:1347] Generated new cluster ID: 26f3f3673921436895a216ff5a57e0b4
I20260929 16:30:38.897773 22521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:38.911996 22521 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:38.913394 22521 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:38.925844 22521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ba6a28de51240cbacc0597226d6a9a9: Generated new TSK 0
I20260929 16:30:38.926760 22521 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:38.933012 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 22432
2026-09-29T16:30:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1468 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:38 dist-test-slave-vq9v krb5kdc[22531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:38 dist-test-slave-vq9v krb5kdc[22531](info): set up 2 sockets
Sep 29 16:30:38 dist-test-slave-vq9v krb5kdc[22531](info): commencing operation
krb5kdc: starting...
W20260929 16:30:39.985685 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:40 dist-test-slave-vq9v krb5kdc[22531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:30:40.114784 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44299
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:40.232455 22544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:40.232743 22544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:40.232787 22544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:40.236785 22544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:40.236858 22544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:40.236878 22544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:40.236896 22544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:40.236910 22544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:40.242388 22544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:40.243453 22544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:40.244413 22544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:40.253041 22550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:40.253037 22552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:40.253082 22544 server_base.cc:1034] running on GCE node
W20260929 16:30:40.253347 22549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:40.254004 22544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:40.255205 22544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:40.256410 22544 hybrid_clock.cc:648] HybridClock initialized: now 1790699440256383 us; error 43 us; skew 500 ppm
Sep 29 16:30:40 dist-test-slave-vq9v krb5kdc[22531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:40.259297 22544 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:30:40.260309 22544 webserver.cc:460] Webserver started at http://127.0.98.62:45437/ using document root <none> and password file <none>
I20260929 16:30:40.260926 22544 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:40.261021 22544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:40.261319 22544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:40.263567 22544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "7be78a1ab7814609be68ddee7e82872a"
format_stamp: "Formatted at 2026-09-29 16:30:40 on dist-test-slave-vq9v"
server_key: "39450ead08526278ae091a1a7622c95a"
server_key_iv: "de254386bee792b837ac5afc570af42e"
server_key_version: "encryptionkey@0"
I20260929 16:30:40.264186 22544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "7be78a1ab7814609be68ddee7e82872a"
format_stamp: "Formatted at 2026-09-29 16:30:40 on dist-test-slave-vq9v"
server_key: "39450ead08526278ae091a1a7622c95a"
server_key_iv: "de254386bee792b837ac5afc570af42e"
server_key_version: "encryptionkey@0"
I20260929 16:30:40.268163 22544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 16:30:40.271126 22559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:40.272521 22544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:30:40.272634 22544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "7be78a1ab7814609be68ddee7e82872a"
format_stamp: "Formatted at 2026-09-29 16:30:40 on dist-test-slave-vq9v"
server_key: "39450ead08526278ae091a1a7622c95a"
server_key_iv: "de254386bee792b837ac5afc570af42e"
server_key_version: "encryptionkey@0"
I20260929 16:30:40.272739 22544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:40.277595 22544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:40.281471 22544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:40.281638 22544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:40.288523 22544 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44299
I20260929 16:30:40.288611 22611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44299 every 8 connection(s)
I20260929 16:30:40.289623 22544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:40.291440 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 22544
I20260929 16:30:40.291548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:40.291957 392 external_mini_cluster.cc:1428] Setting key 136f248722784852842330305c08e370
I20260929 16:30:40.292968 22612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:40.297947 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:40.293422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57669 (local address 127.0.98.62:44299)
0929 16:30:40.293860 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:40.293880 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:40.293924 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:30:40.294879 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:40.294903 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:40.295281 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:40.295685 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:40.296408 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:40.296414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:40.296430 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:40.296443 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:40.296508 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:40.296535 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:40.296699 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:40.296775 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 16:30:40.296934 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":117,"threads_started":1}
I20260929 16:30:40.300634 22612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a: Bootstrap starting.
I20260929 16:30:40.300937 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:30:40.303277 22612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:40.304168 22612 log.cc:824] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:40.305009 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:40.303033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37955 (local address 127.0.98.62:44299)
0929 16:30:40.303197 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:40.303205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:40.303401 (+ 196us) server_negotiation.cc:407] Connection header received
0929 16:30:40.303493 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:40.303499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:40.303546 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:40.303650 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:40.304466 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:40.304471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:40.304476 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:30:40.304490 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:40.304531 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:30:40.304553 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:40.304674 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:40.304809 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 16:30:40.304844 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:30:40.306475 22612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a: No bootstrap required, opened a new log
I20260929 16:30:40.310093 22612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be78a1ab7814609be68ddee7e82872a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44299 } }
I20260929 16:30:40.310350 22612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:30:40.310413 22612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7be78a1ab7814609be68ddee7e82872a, State: Initialized, Role: FOLLOWER
I20260929 16:30:40.310906 22612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be78a1ab7814609be68ddee7e82872a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44299 } }
I20260929 16:30:40.311064 22612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:30:40.311139 22612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:30:40.311273 22612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:40.312294 22612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be78a1ab7814609be68ddee7e82872a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44299 } }
I20260929 16:30:40.312660 22612 leader_election.cc:302] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7be78a1ab7814609be68ddee7e82872a; no voters:
I20260929 16:30:40.313014 22612 leader_election.cc:288] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:30:40.313156 22622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:40.313459 22622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a [term 1 LEADER]: Becoming Leader. State: Replica: 7be78a1ab7814609be68ddee7e82872a, State: Running, Role: LEADER
I20260929 16:30:40.313853 22622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be78a1ab7814609be68ddee7e82872a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44299 } }
I20260929 16:30:40.314575 22612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:40.315814 22623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7be78a1ab7814609be68ddee7e82872a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be78a1ab7814609be68ddee7e82872a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44299 } } }
I20260929 16:30:40.315821 22624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7be78a1ab7814609be68ddee7e82872a. Latest consensus state: current_term: 1 leader_uuid: "7be78a1ab7814609be68ddee7e82872a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be78a1ab7814609be68ddee7e82872a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44299 } } }
I20260929 16:30:40.315954 22624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:40.315954 22623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:40.316442 22631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:40.319473 22631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:40.325548 22631 catalog_manager.cc:1347] Generated new cluster ID: b76b95ab386342fc9ad9f851a15c0008
I20260929 16:30:40.325641 22631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:40.339183 22631 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:40.339522 22631 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:40.344933 22631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7be78a1ab7814609be68ddee7e82872a: Generated new TSK 0
I20260929 16:30:40.345773 22631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:30:40.350432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 22544
2026-09-29T16:30:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1417 ms)
[----------] 18 tests from AuthTokenIssuingTest (26222 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:40 dist-test-slave-vq9v krb5kdc[22643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:40 dist-test-slave-vq9v krb5kdc[22643](info): set up 2 sockets
Sep 29 16:30:40 dist-test-slave-vq9v krb5kdc[22643](info): commencing operation
krb5kdc: starting...
W20260929 16:30:41.421682 392 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:41 dist-test-slave-vq9v krb5kdc[22643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:30:41.556609 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33323
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33323,127.0.98.61:40575,127.0.98.60:34345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:41.674117 22656 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:30:41.674458 22656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:41.674511 22656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:41.674594 22656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:41.678957 22656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:41.679036 22656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:41.679056 22656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:41.679075 22656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:41.679090 22656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:41.684881 22656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33323,127.0.98.61:40575,127.0.98.60:34345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:41.686079 22656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:41.687141 22656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:30:41.696275 22656 server_base.cc:1034] running on GCE node
W20260929 16:30:41.696223 22662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:41.696228 22664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:41.696218 22661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:41.697152 22656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:41.698407 22656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:41.699572 22656 hybrid_clock.cc:648] HybridClock initialized: now 1790699441699543 us; error 48 us; skew 500 ppm
Sep 29 16:30:41 dist-test-slave-vq9v krb5kdc[22643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:41.702803 22656 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:30:41.703887 22656 webserver.cc:460] Webserver started at http://127.0.98.62:34495/ using document root <none> and password file <none>
I20260929 16:30:41.704522 22656 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:41.704592 22656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:41.704828 22656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:41.707155 22656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "cb28c7a43d6440149f3cc07aff68245c"
format_stamp: "Formatted at 2026-09-29 16:30:41 on dist-test-slave-vq9v"
server_key: "9cbe9448f0100617d219bcda77f7901c"
server_key_iv: "d890bba5e06e37d4d9c7a86385f59919"
server_key_version: "encryptionkey@0"
I20260929 16:30:41.707808 22656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "cb28c7a43d6440149f3cc07aff68245c"
format_stamp: "Formatted at 2026-09-29 16:30:41 on dist-test-slave-vq9v"
server_key: "9cbe9448f0100617d219bcda77f7901c"
server_key_iv: "d890bba5e06e37d4d9c7a86385f59919"
server_key_version: "encryptionkey@0"
I20260929 16:30:41.712049 22656 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 16:30:41.715003 22671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:41.716405 22656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:30:41.716588 22656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "cb28c7a43d6440149f3cc07aff68245c"
format_stamp: "Formatted at 2026-09-29 16:30:41 on dist-test-slave-vq9v"
server_key: "9cbe9448f0100617d219bcda77f7901c"
server_key_iv: "d890bba5e06e37d4d9c7a86385f59919"
server_key_version: "encryptionkey@0"
I20260929 16:30:41.716729 22656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:41.742537 22656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:41.746270 22656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:41.746527 22656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:41.754490 22656 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33323
I20260929 16:30:41.754818 22723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33323 every 8 connection(s)
I20260929 16:30:41.755600 22656 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:41.758854 22724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:41.763108 22724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33323 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:30:41.764148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 22656
I20260929 16:30:41.764274 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:41.764652 392 external_mini_cluster.cc:1428] Setting key b694be62da3a2c3df83396f05dddba36
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 29 16:30:41 dist-test-slave-vq9v krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:30:41.785545 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:41.766186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33323 (local address 127.0.0.1:39452)
0929 16:30:41.767002 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:41.767079 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:41.768593 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:41.769532 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:41.769542 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:41.770154 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:41.770623 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:41.770652 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:41.771980 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:41.771986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:41.773497 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:41.773512 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:41.774289 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:41.774310 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:41.774433 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:41.775076 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:41.775095 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:41.778640 (+ 3545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:41.781885 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:41.781896 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:41.781906 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:41.782204 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:41.782673 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:41.782678 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:41.782685 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:41.782809 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:41.783520 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:41.783527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:41.783690 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 16:30:41.783887 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 16:30:41.784345 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":124,"thread_start_us":203,"threads_started":1}
I20260929 16:30:41.785545 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:41.767250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39452 (local address 127.0.98.62:33323)
0929 16:30:41.767586 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:41.767598 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:41.767640 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:30:41.768999 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:41.769004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:41.769306 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:41.769471 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:41.770841 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:41.771826 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:41.773679 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:41.774092 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:41.774224 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:41.778904 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:41.778917 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:41.778940 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:41.778981 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:41.781722 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:41.782374 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:41.782386 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:41.782388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:41.782523 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:41.782920 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:41.782926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:41.782928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:41.783382 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:41.783543 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:41.783952 (+ 409us) server_negotiation.cc:299] Negotiation successful
0929 16:30:41.784257 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":126,"threads_started":1}
I20260929 16:30:41.788442 22724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:30:41.789541 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:41.789178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40575 (local address 127.0.0.1:56370)
0929 16:30:41.789386 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:41.789471 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260929 16:30:41.789769 22673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40575: connect: Connection refused (error 111)
I20260929 16:30:41.790764 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:41.790534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40575 (local address 127.0.0.1:56378)
0929 16:30:41.790654 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:41.790706 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 16:30:41.791014 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40575: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40575: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 16:30:41.796124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40575
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33323,127.0.98.61:40575,127.0.98.60:34345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:30:41.840390 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } attempt: 1
I20260929 16:30:41.841506 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:41.841097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40575 (local address 127.0.0.1:56388)
0929 16:30:41.841356 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:41.841422 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 16:30:41.842326 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:41.842103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40575 (local address 127.0.0.1:56398)
0929 16:30:41.842223 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:41.842269 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 16:30:41.842505 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40575: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40575: connect: Connection refused (error 111)
I20260929 16:30:41.910701 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } attempt: 2
I20260929 16:30:41.911638 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:41.911323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40575 (local address 127.0.0.1:56408)
0929 16:30:41.911477 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:41.911559 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 16:30:41.912571 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:41.912335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40575 (local address 127.0.0.1:56422)
0929 16:30:41.912456 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:41.912511 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 16:30:41.912760 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40575: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40575: connect: Connection refused (error 111)
W20260929 16:30:41.916903 22731 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:30:41.917241 22731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:41.917330 22731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:41.917415 22731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:41.921877 22731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:41.921957 22731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:41.921978 22731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:41.921996 22731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:41.922051 22731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:41.927882 22731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33323,127.0.98.61:40575,127.0.98.60:34345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:41.929065 22731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:41.930172 22731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:41.939145 22736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:41.939210 22731 server_base.cc:1034] running on GCE node
W20260929 16:30:41.939138 22737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:41.939138 22739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:41.940053 22731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:41.941340 22731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:41.942525 22731 hybrid_clock.cc:648] HybridClock initialized: now 1790699441942530 us; error 49 us; skew 500 ppm
Sep 29 16:30:41 dist-test-slave-vq9v krb5kdc[22643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:41.945645 22731 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 16:30:41.946719 22731 webserver.cc:460] Webserver started at http://127.0.98.61:42123/ using document root <none> and password file <none>
I20260929 16:30:41.947338 22731 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:41.947436 22731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:41.947674 22731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:41.949952 22731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-1/data/instance:
uuid: "c61a7c3a34314bf5baa2ef5e53898778"
format_stamp: "Formatted at 2026-09-29 16:30:41 on dist-test-slave-vq9v"
server_key: "b0b278c78d998082278051641dea35c7"
server_key_iv: "4cadd03529515eb4009e5dc2c7213489"
server_key_version: "encryptionkey@0"
I20260929 16:30:41.950549 22731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-1/wal/instance:
uuid: "c61a7c3a34314bf5baa2ef5e53898778"
format_stamp: "Formatted at 2026-09-29 16:30:41 on dist-test-slave-vq9v"
server_key: "b0b278c78d998082278051641dea35c7"
server_key_iv: "4cadd03529515eb4009e5dc2c7213489"
server_key_version: "encryptionkey@0"
I20260929 16:30:41.954679 22731 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:30:41.957609 22746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:41.959014 22731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:41.959180 22731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-1/wal
uuid: "c61a7c3a34314bf5baa2ef5e53898778"
format_stamp: "Formatted at 2026-09-29 16:30:41 on dist-test-slave-vq9v"
server_key: "b0b278c78d998082278051641dea35c7"
server_key_iv: "4cadd03529515eb4009e5dc2c7213489"
server_key_version: "encryptionkey@0"
I20260929 16:30:41.959316 22731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:41.980710 22731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:41.984190 22731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:41.984431 22731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:41.991892 22731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40575
I20260929 16:30:41.992198 22798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40575 every 8 connection(s)
I20260929 16:30:41.992925 22731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-1/data/info.pb
I20260929 16:30:41.993764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 22731
I20260929 16:30:41.993911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-1/wal/instance
I20260929 16:30:41.994196 392 external_mini_cluster.cc:1428] Setting key 9a9852eda7b3aaa80daa7b4e37c01fed
I20260929 16:30:41.995910 22799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:41.999876 22799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33323 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260929 16:30:42.003937 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } attempt: 3
Sep 29 16:30:42 dist-test-slave-vq9v krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 16:30:42 dist-test-slave-vq9v krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:30:42.015791 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.004487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40575 (local address 127.0.0.1:56426)
0929 16:30:42.004684 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:42.004700 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:42.004785 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:42.005445 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:42.005448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:42.005632 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:42.005766 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:42.005774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.007064 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.007068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:42.007699 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:42.007717 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.008591 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.008594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:42.008707 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:42.009384 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:42.009403 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:42.011038 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:42.013582 (+ 2544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:42.013587 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:42.013591 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:42.013911 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:42.014330 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:42.014336 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:42.014341 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:42.014445 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:42.015187 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:42.015192 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:42.015283 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 16:30:42.015367 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 16:30:42.015483 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":3}
I20260929 16:30:42.016940 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.001996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39454 (local address 127.0.98.62:33323)
0929 16:30:42.002181 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.002188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.003080 (+ 892us) server_negotiation.cc:407] Connection header received
0929 16:30:42.004369 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.004373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.004435 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.004514 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.007291 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.008136 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.009438 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.009902 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.010044 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:42.012744 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:42.012752 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:42.012754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:42.012776 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:42.015005 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.015556 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.015559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.015561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.015638 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.015943 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.015947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.015948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.016136 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:42.016201 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:42.016685 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 16:30:42.016801 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:30:42.017078 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.002416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33323 (local address 127.0.0.1:39454)
0929 16:30:42.002917 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:42.002960 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:42.004107 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:42.004834 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:42.004846 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:42.005416 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:42.007160 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:42.007167 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.008281 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.008284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:42.009234 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:42.009244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.010123 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.010128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:42.010269 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:42.010897 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:42.010919 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:42.012595 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:42.015133 (+ 2538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:42.015143 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:42.015151 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:42.015453 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:42.015733 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:42.015736 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:42.015737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:42.015825 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:42.016280 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:42.016285 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:42.016415 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 16:30:42.016566 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 16:30:42.016766 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":1237,"spinlock_wait_cycles":27520,"thread_start_us":134,"threads_started":1}
I20260929 16:30:42.017078 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.004574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56426 (local address 127.0.98.61:40575)
0929 16:30:42.005002 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.005010 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.005030 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:30:42.005064 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.005068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.005291 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.005475 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.005923 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.006927 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.007835 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.008449 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.009031 (+ 582us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:42.011213 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:42.011222 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:42.011240 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:42.011272 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:42.013456 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.014027 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.014032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.014038 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.014170 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.014552 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.014555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.014556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.015054 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:42.015232 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:42.015549 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 16:30:42.015915 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":348,"spinlock_wait_cycles":19328,"thread_start_us":95,"threads_started":1}
I20260929 16:30:42.018981 22799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:30:42.019868 22724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:30:42.020658 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.020370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34345 (local address 127.0.0.1:58516)
0929 16:30:42.020487 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:42.020570 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 16:30:42.021441 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.021127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34345 (local address 127.0.0.1:58524)
0929 16:30:42.021320 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:42.021390 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260929 16:30:42.021561 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34345: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34345: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Sep 29 16:30:42 dist-test-slave-vq9v krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 16:30:42.024340 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34345
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33323,127.0.98.61:40575,127.0.98.60:34345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:30:42.028365 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.019599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56432 (local address 127.0.98.61:40575)
0929 16:30:42.019708 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.019713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.019989 (+ 276us) server_negotiation.cc:407] Connection header received
0929 16:30:42.020020 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.020024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.020063 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.020127 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.020643 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.021432 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.022294 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.022520 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.022607 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:42.024861 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:42.024867 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:42.024874 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:42.024895 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:42.026677 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.027159 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.027165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.027167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.027242 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.027514 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.027519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.027521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.027712 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:42.027817 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:42.028075 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 16:30:42.028243 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":35}
I20260929 16:30:42.028365 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.019586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40575 (local address 127.0.0.1:56432)
0929 16:30:42.019776 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:42.019790 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:42.019911 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:42.020139 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:42.020141 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:42.020360 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:42.020442 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:42.020447 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.021587 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.021590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:42.022156 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:42.022164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.022624 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.022627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:42.022759 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:42.023306 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:42.023320 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:42.024713 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:42.026824 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:42.026828 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:42.026830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:42.027035 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:42.027338 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:42.027342 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:42.027344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:42.027426 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:42.027816 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:42.027819 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:42.027910 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 16:30:42.028017 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 16:30:42.028182 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 16:30:42.029493 22799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:30:42.030267 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.029995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34345 (local address 127.0.0.1:58540)
0929 16:30:42.030116 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:42.030205 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 16:30:42.030493 22749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34345: connect: Connection refused (error 111)
I20260929 16:30:42.031436 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.031223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34345 (local address 127.0.0.1:58546)
0929 16:30:42.031332 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:42.031383 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:30:42.031595 22799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34345: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34345: connect: Connection refused (error 111)
I20260929 16:30:42.052774 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } attempt: 1
I20260929 16:30:42.053738 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.053410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34345 (local address 127.0.0.1:58552)
0929 16:30:42.053591 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:42.053661 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 16:30:42.054601 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.054344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34345 (local address 127.0.0.1:58566)
0929 16:30:42.054477 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:42.054533 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 16:30:42.054780 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34345: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34345: connect: Connection refused (error 111)
I20260929 16:30:42.080866 22799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } attempt: 1
I20260929 16:30:42.081811 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.081492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34345 (local address 127.0.0.1:58580)
0929 16:30:42.081663 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:42.081730 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 16:30:42.082569 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.082360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34345 (local address 127.0.0.1:58594)
0929 16:30:42.082466 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:42.082518 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 16:30:42.082703 22799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34345: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34345: connect: Connection refused (error 111)
I20260929 16:30:42.129972 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } attempt: 2
I20260929 16:30:42.130893 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.130569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34345 (local address 127.0.0.1:58596)
0929 16:30:42.130752 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:42.130819 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 16:30:42.131780 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.131550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34345 (local address 127.0.0.1:58608)
0929 16:30:42.131674 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:42.131727 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 16:30:42.131902 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34345: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34345: connect: Connection refused (error 111)
W20260929 16:30:42.140295 22805 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:30:42.140657 22805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:42.140722 22805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:42.140822 22805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:42.145107 22805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:42.145252 22805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:42.145303 22805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:42.145341 22805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:42.145373 22805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:42.150871 22799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } attempt: 2
I20260929 16:30:42.151196 22805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33323,127.0.98.61:40575,127.0.98.60:34345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:42.151795 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.151445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34345 (local address 127.0.0.1:58622)
0929 16:30:42.151626 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:42.151702 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 16:30:42.152546 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.152318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34345 (local address 127.0.0.1:58636)
0929 16:30:42.152428 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:42.152484 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 16:30:42.152689 22799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34345: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34345: connect: Connection refused (error 111)
I20260929 16:30:42.152726 22805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:42.153888 22805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:30:42.162866 22805 server_base.cc:1034] running on GCE node
W20260929 16:30:42.162828 22814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:42.162906 22811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:42.162828 22812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:42.163648 22805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:42.164873 22805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:42.166069 22805 hybrid_clock.cc:648] HybridClock initialized: now 1790699442166046 us; error 41 us; skew 500 ppm
Sep 29 16:30:42 dist-test-slave-vq9v krb5kdc[22643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:42.168911 22805 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 16:30:42.169849 22805 webserver.cc:460] Webserver started at http://127.0.98.60:36495/ using document root <none> and password file <none>
I20260929 16:30:42.170387 22805 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:42.170444 22805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:42.170608 22805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:42.172596 22805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-2/data/instance:
uuid: "e194166d1ecd4bf2bef8fece9dbeab81"
format_stamp: "Formatted at 2026-09-29 16:30:42 on dist-test-slave-vq9v"
server_key: "6d3e138a89e0d5f39f235a4937e1720d"
server_key_iv: "127552b03f1f321c8277925edf4261a2"
server_key_version: "encryptionkey@0"
I20260929 16:30:42.173122 22805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-2/wal/instance:
uuid: "e194166d1ecd4bf2bef8fece9dbeab81"
format_stamp: "Formatted at 2026-09-29 16:30:42 on dist-test-slave-vq9v"
server_key: "6d3e138a89e0d5f39f235a4937e1720d"
server_key_iv: "127552b03f1f321c8277925edf4261a2"
server_key_version: "encryptionkey@0"
I20260929 16:30:42.176880 22805 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:42.179458 22821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:42.180771 22805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:30:42.180876 22805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-2/wal
uuid: "e194166d1ecd4bf2bef8fece9dbeab81"
format_stamp: "Formatted at 2026-09-29 16:30:42 on dist-test-slave-vq9v"
server_key: "6d3e138a89e0d5f39f235a4937e1720d"
server_key_iv: "127552b03f1f321c8277925edf4261a2"
server_key_version: "encryptionkey@0"
I20260929 16:30:42.180966 22805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:42.195508 22805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:42.199137 22805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:42.199332 22805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:42.207842 22805 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34345
I20260929 16:30:42.208192 22873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34345 every 8 connection(s)
I20260929 16:30:42.209146 22805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-2/data/info.pb
I20260929 16:30:42.211205 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 22805
I20260929 16:30:42.211314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790699152396020-392-0/minicluster-data/master-2/wal/instance
I20260929 16:30:42.211592 392 external_mini_cluster.cc:1428] Setting key 471439a0a3caffd9b50970631dcb5827
I20260929 16:30:42.212893 22874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:42.218123 22874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33323 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:30:42 dist-test-slave-vq9v krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 16:30:42 dist-test-slave-vq9v krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:30:42.228462 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.213658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58642 (local address 127.0.98.60:34345)
0929 16:30:42.214114 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.214133 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.214480 (+ 347us) server_negotiation.cc:407] Connection header received
0929 16:30:42.215314 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.215331 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.215668 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.215950 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.216872 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.217946 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.219115 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.219592 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.219769 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:42.222520 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:42.222525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:42.222535 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:42.222562 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:42.224405 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.224849 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.224857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.224865 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.225012 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.225422 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.225425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.225427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.225937 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:42.226113 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:42.226910 (+ 797us) server_negotiation.cc:299] Negotiation successful
0929 16:30:42.227244 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":136,"threads_started":1}
I20260929 16:30:42.230458 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.220155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33323 (local address 127.0.0.1:39460)
0929 16:30:42.220584 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:42.220630 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:42.220791 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:42.221090 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:42.221095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:42.221638 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:42.221772 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:42.221778 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.222598 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.222601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:42.223248 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:42.223256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.223737 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.223740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:42.223849 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:42.224324 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:42.224336 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:42.225907 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:42.228447 (+ 2540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:42.228457 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:42.228466 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:42.228805 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:42.229132 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:42.229136 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:42.229139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:42.229266 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:42.229795 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:42.229801 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:42.229921 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 16:30:42.230069 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 16:30:42.230256 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":179,"threads_started":1}
I20260929 16:30:42.230458 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.220226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39460 (local address 127.0.98.62:33323)
0929 16:30:42.220399 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.220404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.220713 (+ 309us) server_negotiation.cc:407] Connection header received
0929 16:30:42.220886 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.220889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.220944 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.221041 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.221897 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.222465 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.223374 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.223623 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.223738 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:42.226074 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:42.226077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:42.226080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:42.226108 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:42.228330 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.228932 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.228935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.228937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.229032 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.229393 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.229396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.229398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.229628 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:42.229744 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:42.230178 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 16:30:42.230300 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:30:42.231034 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } attempt: 3
I20260929 16:30:42.233594 22874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:30:42 dist-test-slave-vq9v krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 16:30:42 dist-test-slave-vq9v krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 16:30:42 dist-test-slave-vq9v krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 16:30:42.242274 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.231411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34345 (local address 127.0.0.1:58652)
0929 16:30:42.231576 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:42.231592 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:42.231675 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:42.232487 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:42.232491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:42.232687 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:42.232791 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:42.232798 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.235455 (+ 2657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.235460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:42.236148 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:42.236157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.236643 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.236646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:42.236729 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:42.237264 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:42.237281 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:42.238790 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:42.240713 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:42.240718 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:42.240721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:42.240935 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:42.241241 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:42.241245 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:42.241248 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:42.241343 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:42.241734 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:42.241737 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:42.241844 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 16:30:42.241956 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 16:30:42.242113 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 16:30:42.242820 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.231820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58652 (local address 127.0.98.60:34345)
0929 16:30:42.232262 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.232266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.232279 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:30:42.232318 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.232321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.232363 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.232446 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.234608 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.235314 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.236270 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.236522 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.236655 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:42.238918 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:42.238921 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:42.238924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:42.238953 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:42.240602 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.241032 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.241038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.241040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.241128 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.241436 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.241441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.241443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.241635 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:42.241745 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:42.242510 (+ 765us) server_negotiation.cc:299] Negotiation successful
0929 16:30:42.242648 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364}
I20260929 16:30:42.243896 22799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } attempt: 3
I20260929 16:30:42.244155 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.234448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56442 (local address 127.0.98.61:40575)
0929 16:30:42.234612 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.234617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.234634 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:30:42.234679 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.234682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.234730 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.234811 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.235809 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.236664 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.237571 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.237851 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.237951 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:42.240051 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:42.240055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:42.240057 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:42.240093 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:42.241845 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.242678 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.242681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.242683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.242782 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.243217 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.243221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.243222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.243470 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:42.243543 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:42.243885 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 16:30:42.244012 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:30:42.244280 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.231713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39476 (local address 127.0.98.62:33323)
0929 16:30:42.231921 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.231925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.231942 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:30:42.231969 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.231971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.232005 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.232096 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.232884 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.233510 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.234573 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.234899 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.234995 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:42.238702 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:42.238706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:42.238710 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:42.238735 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:42.241416 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.241951 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.241955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.241956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.242032 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.242369 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.242373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.242376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.242579 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:42.242827 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:42.244004 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0929 16:30:42.244134 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260929 16:30:42.245509 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.234096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40575 (local address 127.0.0.1:56442)
0929 16:30:42.234343 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:42.234358 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:42.234426 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:42.234928 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:42.234931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:42.235129 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:42.235268 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:42.235275 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.236827 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.236831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:42.237447 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:42.237456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.237952 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.237956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:42.238048 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:42.238509 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:42.238532 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:42.239854 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:42.241972 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:42.241975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:42.241977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:42.242249 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:42.242999 (+ 750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:42.243002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:42.243004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:42.243107 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:42.243625 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:42.243627 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:42.243704 (+ 77us) client_negotiation.cc:769] Sending connection context
0929 16:30:42.243813 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 16:30:42.245358 (+ 1545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260929 16:30:42.246919 22724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c: Bootstrap starting.
Sep 29 16:30:42 dist-test-slave-vq9v krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 16:30:42.250186 22724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c: Neither blocks nor log segments found. Creating new log.
Sep 29 16:30:42 dist-test-slave-vq9v krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 16:30:42.251260 22724 log.cc:824] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:42.252413 22874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:30:42.253791 22724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c: No bootstrap required, opened a new log
I20260929 16:30:42.253906 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.244627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58662 (local address 127.0.98.60:34345)
0929 16:30:42.244739 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.244746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.244761 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:30:42.244796 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.244798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.244838 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.245378 (+ 540us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.245480 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.246176 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.247253 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.247530 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.247987 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:42.250110 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:42.250116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:42.250119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:42.250147 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:42.251825 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.252348 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.252351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.252352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.252423 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.252721 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.252726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.252728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.252964 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:42.253533 (+ 569us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:42.253628 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 16:30:42.253740 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:30:42.253976 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.244321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34345 (local address 127.0.0.1:58662)
0929 16:30:42.244604 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:42.244620 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:42.244699 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:42.244966 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:42.244969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:42.245133 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:42.245270 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:42.245277 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.246306 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.246310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:42.247060 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:42.247070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.247675 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.247678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:42.247758 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:42.248300 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:42.248318 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:42.249923 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:42.251938 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:42.251941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:42.251943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:42.252244 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:42.252530 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:42.252535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:42.252537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:42.252627 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:42.253079 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:42.253086 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:42.253192 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 16:30:42.253339 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 16:30:42.253548 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20260929 16:30:42.256702 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.245664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56450 (local address 127.0.98.61:40575)
0929 16:30:42.245754 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.245759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.245924 (+ 165us) server_negotiation.cc:407] Connection header received
0929 16:30:42.246118 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.246120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.246167 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.246249 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.246865 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.247439 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.248342 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.248574 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.248688 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:42.251500 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:42.251503 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:42.251504 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:42.251526 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:42.254417 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.254940 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.254944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.254946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.255032 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.255323 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.255326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.255331 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.255533 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:42.255754 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:42.256415 (+ 661us) server_negotiation.cc:299] Negotiation successful
0929 16:30:42.256529 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 16:30:42.257462 22799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c61a7c3a34314bf5baa2ef5e53898778: Bootstrap starting.
I20260929 16:30:42.258504 22724 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb28c7a43d6440149f3cc07aff68245c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33323 } } peers { permanent_uuid: "c61a7c3a34314bf5baa2ef5e53898778" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } } peers { permanent_uuid: "e194166d1ecd4bf2bef8fece9dbeab81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } }
I20260929 16:30:42.258769 22724 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb28c7a43d6440149f3cc07aff68245c, State: Initialized, Role: FOLLOWER
I20260929 16:30:42.259146 22724 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb28c7a43d6440149f3cc07aff68245c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33323 } } peers { permanent_uuid: "c61a7c3a34314bf5baa2ef5e53898778" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } } peers { permanent_uuid: "e194166d1ecd4bf2bef8fece9dbeab81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } }
I20260929 16:30:42.260586 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:30:42.260614 22799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c61a7c3a34314bf5baa2ef5e53898778: Neither blocks nor log segments found. Creating new log.
Sep 29 16:30:42 dist-test-slave-vq9v krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 16:30:42.260700 22724 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:42.261662 22799 log.cc:824] T 00000000000000000000000000000000 P c61a7c3a34314bf5baa2ef5e53898778: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:42.262115 22698 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "cb28c7a43d6440149f3cc07aff68245c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39476
I20260929 16:30:42.262391 22698 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 16:30:42.262523 22698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:42.263996 22698 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb28c7a43d6440149f3cc07aff68245c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33323 } } peers { permanent_uuid: "c61a7c3a34314bf5baa2ef5e53898778" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } } peers { permanent_uuid: "e194166d1ecd4bf2bef8fece9dbeab81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } }
I20260929 16:30:42.265388 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.252909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34345 (local address 127.0.0.1:58676)
0929 16:30:42.253038 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:42.253054 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:42.253130 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:42.255131 (+ 2001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:42.255134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:42.255302 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:42.255413 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:42.255422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.256498 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.256501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:42.257266 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:42.257276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.258961 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.258965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:42.259060 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:42.259585 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:42.259598 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:42.261169 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:42.263747 (+ 2578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:42.263751 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:42.263753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:42.263990 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:42.264365 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:42.264368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:42.264370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:42.264449 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:42.264844 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:42.264847 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:42.264926 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 16:30:42.265001 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 16:30:42.265175 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 16:30:42.265661 22881 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb28c7a43d6440149f3cc07aff68245c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33323 } } peers { permanent_uuid: "c61a7c3a34314bf5baa2ef5e53898778" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } } peers { permanent_uuid: "e194166d1ecd4bf2bef8fece9dbeab81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } } }
I20260929 16:30:42.266283 22881 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:30:42.265736 22698 leader_election.cc:288] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c [CANDIDATE]: Term 1 election: Requested vote from peers c61a7c3a34314bf5baa2ef5e53898778 (127.0.98.61:40575), e194166d1ecd4bf2bef8fece9dbeab81 (127.0.98.60:34345)
I20260929 16:30:42.266624 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.253050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.0.98.60:34345)
0929 16:30:42.253798 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.253807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.254919 (+ 1112us) server_negotiation.cc:407] Connection header received
0929 16:30:42.254960 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.254963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.255007 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.255516 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.255585 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.256390 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.257666 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.258695 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.259405 (+ 710us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:42.261427 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:42.261431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:42.261444 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:42.261471 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:42.263573 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.264167 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.264172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.264174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.264265 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.264540 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.264542 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.264544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.264748 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:42.266049 (+ 1301us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:42.266135 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 16:30:42.266395 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":162,"threads_started":1}
I20260929 16:30:42.270244 22799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c61a7c3a34314bf5baa2ef5e53898778: No bootstrap required, opened a new log
W20260929 16:30:42.270488 22892 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:30:42.270567 22892 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:30:42.271252 22874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e194166d1ecd4bf2bef8fece9dbeab81: Bootstrap starting.
I20260929 16:30:42.273561 22799 raft_consensus.cc:348] T 00000000000000000000000000000000 P c61a7c3a34314bf5baa2ef5e53898778 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb28c7a43d6440149f3cc07aff68245c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33323 } } peers { permanent_uuid: "c61a7c3a34314bf5baa2ef5e53898778" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } } peers { permanent_uuid: "e194166d1ecd4bf2bef8fece9dbeab81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } }
I20260929 16:30:42.273833 22799 raft_consensus.cc:729] T 00000000000000000000000000000000 P c61a7c3a34314bf5baa2ef5e53898778 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c61a7c3a34314bf5baa2ef5e53898778, State: Initialized, Role: FOLLOWER
I20260929 16:30:42.274187 22799 consensus_queue.cc:260] T 00000000000000000000000000000000 P c61a7c3a34314bf5baa2ef5e53898778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb28c7a43d6440149f3cc07aff68245c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33323 } } peers { permanent_uuid: "c61a7c3a34314bf5baa2ef5e53898778" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } } peers { permanent_uuid: "e194166d1ecd4bf2bef8fece9dbeab81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } }
I20260929 16:30:42.274950 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P c61a7c3a34314bf5baa2ef5e53898778 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb28c7a43d6440149f3cc07aff68245c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33323 } } peers { permanent_uuid: "c61a7c3a34314bf5baa2ef5e53898778" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } } peers { permanent_uuid: "e194166d1ecd4bf2bef8fece9dbeab81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } } }
I20260929 16:30:42.275290 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P c61a7c3a34314bf5baa2ef5e53898778 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:30:42.275763 22799 sys_catalog.cc:564] T 00000000000000000000000000000000 P c61a7c3a34314bf5baa2ef5e53898778 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:42.275969 22773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cb28c7a43d6440149f3cc07aff68245c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c61a7c3a34314bf5baa2ef5e53898778"
I20260929 16:30:42.276217 22773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c61a7c3a34314bf5baa2ef5e53898778 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:42.273274 22656 master_runner.cc:186] Error getting master registration for 127.0.98.61:40575: OK, instance_id { permanent_uuid: "c61a7c3a34314bf5baa2ef5e53898778" instance_seqno: 1790699441989858 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:30:42.277230 22656 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:30:42.277769 22773 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c61a7c3a34314bf5baa2ef5e53898778 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb28c7a43d6440149f3cc07aff68245c in term 1.
I20260929 16:30:42.278298 22673 leader_election.cc:302] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c61a7c3a34314bf5baa2ef5e53898778, cb28c7a43d6440149f3cc07aff68245c; no voters:
I20260929 16:30:42.278635 22881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:42.278882 22881 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c [term 1 LEADER]: Becoming Leader. State: Replica: cb28c7a43d6440149f3cc07aff68245c, State: Running, Role: LEADER
I20260929 16:30:42.279119 22874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e194166d1ecd4bf2bef8fece9dbeab81: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:42.279265 22881 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb28c7a43d6440149f3cc07aff68245c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33323 } } peers { permanent_uuid: "c61a7c3a34314bf5baa2ef5e53898778" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } } peers { permanent_uuid: "e194166d1ecd4bf2bef8fece9dbeab81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } }
I20260929 16:30:42.280097 22874 log.cc:824] T 00000000000000000000000000000000 P e194166d1ecd4bf2bef8fece9dbeab81: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:42.281347 22881 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb28c7a43d6440149f3cc07aff68245c. Latest consensus state: current_term: 1 leader_uuid: "cb28c7a43d6440149f3cc07aff68245c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb28c7a43d6440149f3cc07aff68245c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33323 } } peers { permanent_uuid: "c61a7c3a34314bf5baa2ef5e53898778" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } } peers { permanent_uuid: "e194166d1ecd4bf2bef8fece9dbeab81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } } }
I20260929 16:30:42.281483 22881 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:42.282680 22874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e194166d1ecd4bf2bef8fece9dbeab81: No bootstrap required, opened a new log
W20260929 16:30:42.284224 22913 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c61a7c3a34314bf5baa2ef5e53898778: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:30:42.284341 22913 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:30:42.284734 22917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:42.285852 22917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:42.287868 22874 raft_consensus.cc:348] T 00000000000000000000000000000000 P e194166d1ecd4bf2bef8fece9dbeab81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb28c7a43d6440149f3cc07aff68245c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33323 } } peers { permanent_uuid: "c61a7c3a34314bf5baa2ef5e53898778" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } } peers { permanent_uuid: "e194166d1ecd4bf2bef8fece9dbeab81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } }
I20260929 16:30:42.288153 22874 raft_consensus.cc:729] T 00000000000000000000000000000000 P e194166d1ecd4bf2bef8fece9dbeab81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e194166d1ecd4bf2bef8fece9dbeab81, State: Initialized, Role: FOLLOWER
I20260929 16:30:42.288591 22874 consensus_queue.cc:260] T 00000000000000000000000000000000 P e194166d1ecd4bf2bef8fece9dbeab81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb28c7a43d6440149f3cc07aff68245c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33323 } } peers { permanent_uuid: "c61a7c3a34314bf5baa2ef5e53898778" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } } peers { permanent_uuid: "e194166d1ecd4bf2bef8fece9dbeab81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } }
I20260929 16:30:42.290195 22874 sys_catalog.cc:564] T 00000000000000000000000000000000 P e194166d1ecd4bf2bef8fece9dbeab81 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:42.290275 22848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cb28c7a43d6440149f3cc07aff68245c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e194166d1ecd4bf2bef8fece9dbeab81"
I20260929 16:30:42.290520 22848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e194166d1ecd4bf2bef8fece9dbeab81 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:42.287808 22731 master_runner.cc:186] Error getting master registration for 127.0.98.60:34345: OK, instance_id { permanent_uuid: "e194166d1ecd4bf2bef8fece9dbeab81" instance_seqno: 1790699442205050 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:30:42.291796 22731 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:30:42.291834 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.268603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58684 (local address 127.0.98.60:34345)
0929 16:30:42.268717 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.268722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.275122 (+ 6400us) server_negotiation.cc:407] Connection header received
0929 16:30:42.275610 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.275614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.275681 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.275785 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.277860 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.278720 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.281848 (+ 3128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.282187 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.282268 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:42.286925 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:42.286931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:42.286934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:42.286969 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:42.289109 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.290030 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.290034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.290036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.290131 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.290833 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.290837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.290838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.291079 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:42.291238 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:42.291495 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 16:30:42.291647 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":47}
I20260929 16:30:42.292461 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.273988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56460 (local address 127.0.98.61:40575)
0929 16:30:42.274107 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.274111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.275377 (+ 1266us) server_negotiation.cc:407] Connection header received
0929 16:30:42.275696 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.275699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.275758 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.275813 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.278867 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.279497 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.282425 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.282753 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.282836 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:42.286931 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:42.286935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:42.286938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:42.286966 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:42.289137 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.289681 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.289687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.289690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.289800 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.290195 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.290199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.290200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.290439 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:42.290562 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:42.292124 (+ 1562us) server_negotiation.cc:299] Negotiation successful
0929 16:30:42.292257 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 16:30:42.292562 22848 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e194166d1ecd4bf2bef8fece9dbeab81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb28c7a43d6440149f3cc07aff68245c in term 1.
I20260929 16:30:42.293376 22919 sys_catalog.cc:455] T 00000000000000000000000000000000 P e194166d1ecd4bf2bef8fece9dbeab81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb28c7a43d6440149f3cc07aff68245c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33323 } } peers { permanent_uuid: "c61a7c3a34314bf5baa2ef5e53898778" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } } peers { permanent_uuid: "e194166d1ecd4bf2bef8fece9dbeab81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } } }
I20260929 16:30:42.293509 22919 sys_catalog.cc:458] T 00000000000000000000000000000000 P e194166d1ecd4bf2bef8fece9dbeab81 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 16:30:42.299976 22930 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e194166d1ecd4bf2bef8fece9dbeab81: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:30:42.300165 22930 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:30:42.302686 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:42.273695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.0.98.62:33323)
0929 16:30:42.273845 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.273851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.293469 (+ 19618us) server_negotiation.cc:407] Connection header received
0929 16:30:42.293606 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.293612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.293674 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.293730 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.294830 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.295499 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.296488 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:42.296804 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:42.296881 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:42.298791 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:42.298796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:42.298799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:42.298828 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:42.300575 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.301109 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.301113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.301115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.301229 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:42.301579 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:42.301583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:42.301585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:42.301844 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:42.301962 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:42.302392 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 16:30:42.302521 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:30:42.316388 22848 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e194166d1ecd4bf2bef8fece9dbeab81 [term 1 FOLLOWER]: Refusing update from remote peer cb28c7a43d6440149f3cc07aff68245c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:42.317193 22773 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c61a7c3a34314bf5baa2ef5e53898778 [term 1 FOLLOWER]: Refusing update from remote peer cb28c7a43d6440149f3cc07aff68245c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:42.317998 22914 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e194166d1ecd4bf2bef8fece9dbeab81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:42.318895 22914 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c61a7c3a34314bf5baa2ef5e53898778" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:42.322932 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P c61a7c3a34314bf5baa2ef5e53898778 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb28c7a43d6440149f3cc07aff68245c. Latest consensus state: current_term: 1 leader_uuid: "cb28c7a43d6440149f3cc07aff68245c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb28c7a43d6440149f3cc07aff68245c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33323 } } peers { permanent_uuid: "c61a7c3a34314bf5baa2ef5e53898778" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } } peers { permanent_uuid: "e194166d1ecd4bf2bef8fece9dbeab81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } } }
I20260929 16:30:42.323068 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P c61a7c3a34314bf5baa2ef5e53898778 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:30:42.326324 22919 sys_catalog.cc:455] T 00000000000000000000000000000000 P e194166d1ecd4bf2bef8fece9dbeab81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb28c7a43d6440149f3cc07aff68245c. Latest consensus state: current_term: 1 leader_uuid: "cb28c7a43d6440149f3cc07aff68245c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb28c7a43d6440149f3cc07aff68245c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33323 } } peers { permanent_uuid: "c61a7c3a34314bf5baa2ef5e53898778" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } } peers { permanent_uuid: "e194166d1ecd4bf2bef8fece9dbeab81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } } }
I20260929 16:30:42.326464 22919 sys_catalog.cc:458] T 00000000000000000000000000000000 P e194166d1ecd4bf2bef8fece9dbeab81 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:30:42.332731 22914 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb28c7a43d6440149f3cc07aff68245c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb28c7a43d6440149f3cc07aff68245c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33323 } } peers { permanent_uuid: "c61a7c3a34314bf5baa2ef5e53898778" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } } peers { permanent_uuid: "e194166d1ecd4bf2bef8fece9dbeab81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } } }
I20260929 16:30:42.332885 22914 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:42.333014 22932 mvcc.cc:205] Tried to move back new op lower bound from 7334704915720548352 to 7334704915580018688. Current Snapshot: MvccSnapshot[applied={T|T < 7334704915720548352}]
I20260929 16:30:42.333482 22881 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb28c7a43d6440149f3cc07aff68245c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb28c7a43d6440149f3cc07aff68245c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33323 } } peers { permanent_uuid: "c61a7c3a34314bf5baa2ef5e53898778" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } } peers { permanent_uuid: "e194166d1ecd4bf2bef8fece9dbeab81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } } }
I20260929 16:30:42.333581 22881 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:42.334924 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P c61a7c3a34314bf5baa2ef5e53898778 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb28c7a43d6440149f3cc07aff68245c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb28c7a43d6440149f3cc07aff68245c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33323 } } peers { permanent_uuid: "c61a7c3a34314bf5baa2ef5e53898778" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } } peers { permanent_uuid: "e194166d1ecd4bf2bef8fece9dbeab81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } } }
I20260929 16:30:42.335039 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P c61a7c3a34314bf5baa2ef5e53898778 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:30:42.336520 22919 sys_catalog.cc:455] T 00000000000000000000000000000000 P e194166d1ecd4bf2bef8fece9dbeab81 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb28c7a43d6440149f3cc07aff68245c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb28c7a43d6440149f3cc07aff68245c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33323 } } peers { permanent_uuid: "c61a7c3a34314bf5baa2ef5e53898778" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40575 } } peers { permanent_uuid: "e194166d1ecd4bf2bef8fece9dbeab81" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34345 } } }
I20260929 16:30:42.336653 22919 sys_catalog.cc:458] T 00000000000000000000000000000000 P e194166d1ecd4bf2bef8fece9dbeab81 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:30:42.337327 22933 mvcc.cc:205] Tried to move back new op lower bound from 7334704915720548352 to 7334704915580018688. Current Snapshot: MvccSnapshot[applied={T|T < 7334704915720548352}]
I20260929 16:30:42.337790 22935 mvcc.cc:205] Tried to move back new op lower bound from 7334704915720548352 to 7334704915580018688. Current Snapshot: MvccSnapshot[applied={T|T < 7334704915720548352}]
I20260929 16:30:42.342595 22917 catalog_manager.cc:1347] Generated new cluster ID: 974f7d087fdd4a839092249941464ad4
I20260929 16:30:42.342687 22917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:42.368878 22917 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:42.370257 22917 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:42.379594 22917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb28c7a43d6440149f3cc07aff68245c: Generated new TSK 0
I20260929 16:30:42.380604 22917 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:30:42.427326 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:42.424105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39492 (local address 127.0.98.62:33323)
0929 16:30:42.424238 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.424243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.426164 (+ 1921us) server_negotiation.cc:407] Connection header received
0929 16:30:42.426376 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.426380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.426437 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.426495 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.427183 (+ 688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39492: BlockingRecv error: recv got EOF from 127.0.0.1:39492 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 16:30:42.428927 22880 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:42.424931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58690 (local address 127.0.98.60:34345)
0929 16:30:42.425079 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.425084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.427871 (+ 2787us) server_negotiation.cc:407] Connection header received
0929 16:30:42.428006 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.428009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.428066 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.428162 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.428790 (+ 628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58690: BlockingRecv error: recv got EOF from 127.0.0.1:58690 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 16:30:42.429334 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:42.425554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56474 (local address 127.0.98.61:40575)
0929 16:30:42.425666 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.425670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.428239 (+ 2569us) server_negotiation.cc:407] Connection header received
0929 16:30:42.428436 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.428440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.428502 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.428562 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.429129 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56474: BlockingRecv error: recv got EOF from 127.0.0.1:56474 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 16:30:42.434609 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:42.432920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56488 (local address 127.0.98.61:40575)
0929 16:30:42.433031 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.433035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.433553 (+ 518us) server_negotiation.cc:407] Connection header received
0929 16:30:42.433792 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.433796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.433871 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.433927 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.434490 (+ 563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56488: BlockingRecv error: recv got EOF from 127.0.0.1:56488 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 16:30:42.440189 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:42.438418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56504 (local address 127.0.98.61:40575)
0929 16:30:42.438541 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.438546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.439154 (+ 608us) server_negotiation.cc:407] Connection header received
0929 16:30:42.439382 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.439386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.439442 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.439496 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.440078 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56504: BlockingRecv error: recv got EOF from 127.0.0.1:56504 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 16:30:42.446496 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:42.444786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56508 (local address 127.0.98.61:40575)
0929 16:30:42.444930 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.444934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.445508 (+ 574us) server_negotiation.cc:407] Connection header received
0929 16:30:42.445726 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.445730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.445784 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.445839 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.446378 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56508: BlockingRecv error: recv got EOF from 127.0.0.1:56508 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 16:30:42.455140 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:42.453345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56514 (local address 127.0.98.61:40575)
0929 16:30:42.453496 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.453500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.454080 (+ 580us) server_negotiation.cc:407] Connection header received
0929 16:30:42.454315 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.454319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.454376 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.454433 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.455020 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56514: BlockingRecv error: recv got EOF from 127.0.0.1:56514 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 16:30:42.467888 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:42.466040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56530 (local address 127.0.98.61:40575)
0929 16:30:42.466173 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.466178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.466813 (+ 635us) server_negotiation.cc:407] Connection header received
0929 16:30:42.467041 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.467046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.467105 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.467162 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.467768 (+ 606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56530: BlockingRecv error: recv got EOF from 127.0.0.1:56530 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 16:30:42.488505 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:42.486666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56546 (local address 127.0.98.61:40575)
0929 16:30:42.486805 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.486810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.487440 (+ 630us) server_negotiation.cc:407] Connection header received
0929 16:30:42.487665 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.487669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.487736 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.487790 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.488362 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56546: BlockingRecv error: recv got EOF from 127.0.0.1:56546 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 16:30:42.524665 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:42.523401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56550 (local address 127.0.98.61:40575)
0929 16:30:42.523532 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.523536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.523793 (+ 257us) server_negotiation.cc:407] Connection header received
0929 16:30:42.523899 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.523903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.523965 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.524058 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.524538 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56550: BlockingRecv error: recv got EOF from 127.0.0.1:56550 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 16:30:42.592576 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:42.591022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.0.98.61:40575)
0929 16:30:42.591374 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.591378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.591629 (+ 251us) server_negotiation.cc:407] Connection header received
0929 16:30:42.591751 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.591754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.591805 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.591895 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.592476 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56560: BlockingRecv error: recv got EOF from 127.0.0.1:56560 (error 108)
Metrics: {"server-negotiator.queue_time_us":273}
W20260929 16:30:42.724263 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:42.723174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56564 (local address 127.0.98.61:40575)
0929 16:30:42.723319 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.723322 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.723360 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:30:42.723507 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.723509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.723557 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.723641 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.724156 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56564: BlockingRecv error: recv got EOF from 127.0.0.1:56564 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 16:30:42.983588 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:42.982607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56566 (local address 127.0.98.61:40575)
0929 16:30:42.982747 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:42.982752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:42.982850 (+ 98us) server_negotiation.cc:407] Connection header received
0929 16:30:42.982964 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:42.982966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:42.983016 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:42.983085 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:42.983492 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56566: BlockingRecv error: recv got EOF from 127.0.0.1:56566 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:30:43.285828 22913 catalog_manager.cc:1259] Loaded cluster ID: 974f7d087fdd4a839092249941464ad4
I20260929 16:30:43.285916 22913 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c61a7c3a34314bf5baa2ef5e53898778: loading cluster ID for follower catalog manager: success
I20260929 16:30:43.287606 22913 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c61a7c3a34314bf5baa2ef5e53898778: acquiring CA information for follower catalog manager: success
I20260929 16:30:43.288784 22913 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c61a7c3a34314bf5baa2ef5e53898778: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:30:43.301544 22930 catalog_manager.cc:1259] Loaded cluster ID: 974f7d087fdd4a839092249941464ad4
I20260929 16:30:43.301627 22930 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e194166d1ecd4bf2bef8fece9dbeab81: loading cluster ID for follower catalog manager: success
I20260929 16:30:43.303432 22930 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e194166d1ecd4bf2bef8fece9dbeab81: acquiring CA information for follower catalog manager: success
I20260929 16:30:43.304648 22930 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e194166d1ecd4bf2bef8fece9dbeab81: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:30:43.502063 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:43.497747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56580 (local address 127.0.98.61:40575)
0929 16:30:43.497982 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:43.497986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:43.498003 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:30:43.498201 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:43.498205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:43.498336 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:43.498454 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:30:43.498713 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:43.499263 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:43.500147 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:43.500396 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:43.500552 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:43.500926 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:30:43.501462 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:30:43.501610 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:43.501763 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 16:30:43.501832 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":92,"threads_started":1}
I20260929 16:30:43.508881 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:43.505270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.0.98.60:34345)
0929 16:30:43.505507 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:43.505511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:43.505567 (+ 56us) server_negotiation.cc:407] Connection header received
0929 16:30:43.505704 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:43.505706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:43.505846 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:43.505950 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:30:43.506159 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:43.506664 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:43.507515 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:43.507725 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:43.507810 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:43.508045 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:30:43.508473 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:30:43.508569 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:43.508683 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 16:30:43.508737 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":85,"threads_started":1}
I20260929 16:30:43.516124 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:43.512008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58706 (local address 127.0.98.60:34345)
0929 16:30:43.512530 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:43.512534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:43.512563 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:30:43.512608 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:43.512611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:43.512660 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:43.512746 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:30:43.512990 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:43.513789 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:43.514833 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:43.515070 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:43.515176 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:43.515430 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:30:43.515508 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:30:43.515885 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:43.515923 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 16:30:43.515982 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452}
I20260929 16:30:43.516480 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:43.512233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56584 (local address 127.0.98.61:40575)
0929 16:30:43.512409 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:43.512415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:43.512437 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:30:43.512611 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:43.512614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:43.512660 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:43.512746 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:30:43.513194 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:43.513968 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:43.515103 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:43.515486 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:43.515585 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:43.516007 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:30:43.516092 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:30:43.516176 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:43.516301 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 16:30:43.516363 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:30:43.517347 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:43.511947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39506 (local address 127.0.98.62:33323)
0929 16:30:43.512420 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:43.512427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:43.512873 (+ 446us) server_negotiation.cc:407] Connection header received
0929 16:30:43.512980 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:43.512985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:43.513165 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:43.513305 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:30:43.513537 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:43.514233 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:43.515335 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:43.515673 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:43.515783 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:43.516109 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:30:43.516729 (+ 620us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:30:43.516820 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:43.517116 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 16:30:43.517177 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":82,"threads_started":1}
I20260929 16:30:43.520785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 22656
I20260929 16:30:43.527347 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 22731
I20260929 16:30:43.533244 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 22805
2026-09-29T16:30:43Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3183 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:30:43 dist-test-slave-vq9v krb5kdc[23022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:30:43 dist-test-slave-vq9v krb5kdc[23022](info): set up 2 sockets
Sep 29 16:30:43 dist-test-slave-vq9v krb5kdc[23022](info): commencing operation
krb5kdc: starting...
W20260929 16:30:44.577805 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/test-user.keytab.
Sep 29 16:30:44 dist-test-slave-vq9v krb5kdc[23022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:30:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:30:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 16:30:44.706137 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38241
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38241,127.0.98.61:40295,127.0.98.60:38783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:30:44.822083 23035 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:30:44.822449 23035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:44.822522 23035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:44.822602 23035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:44.826807 23035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:44.826905 23035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:44.826941 23035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:44.826978 23035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:44.827009 23035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:44.832696 23035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38241,127.0.98.61:40295,127.0.98.60:38783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:44.833940 23035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:44.834995 23035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:44.843907 23041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:44.843921 23040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:44.843925 23043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:44.844053 23035 server_base.cc:1034] running on GCE node
I20260929 16:30:44.844861 23035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:44.846104 23035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:44.847345 23035 hybrid_clock.cc:648] HybridClock initialized: now 1790699444847326 us; error 37 us; skew 500 ppm
Sep 29 16:30:44 dist-test-slave-vq9v krb5kdc[23022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:44.850612 23035 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 16:30:44.851641 23035 webserver.cc:460] Webserver started at http://127.0.98.62:43855/ using document root <none> and password file <none>
I20260929 16:30:44.852244 23035 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:44.852344 23035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:44.852588 23035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:44.854840 23035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-0/data/instance:
uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d"
format_stamp: "Formatted at 2026-09-29 16:30:44 on dist-test-slave-vq9v"
server_key: "f5b55eb936e8eada7d7cc0c419acaadb"
server_key_iv: "f2a81e4067cb4337bd235aa73b292535"
server_key_version: "encryptionkey@0"
I20260929 16:30:44.855429 23035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-0/wal/instance:
uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d"
format_stamp: "Formatted at 2026-09-29 16:30:44 on dist-test-slave-vq9v"
server_key: "f5b55eb936e8eada7d7cc0c419acaadb"
server_key_iv: "f2a81e4067cb4337bd235aa73b292535"
server_key_version: "encryptionkey@0"
I20260929 16:30:44.859570 23035 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:30:44.862452 23050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:44.864209 23035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:30:44.864370 23035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-0/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-0/wal
uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d"
format_stamp: "Formatted at 2026-09-29 16:30:44 on dist-test-slave-vq9v"
server_key: "f5b55eb936e8eada7d7cc0c419acaadb"
server_key_iv: "f2a81e4067cb4337bd235aa73b292535"
server_key_version: "encryptionkey@0"
I20260929 16:30:44.864531 23035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:44.880041 23035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:44.883875 23035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:44.884099 23035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:44.891985 23035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38241
I20260929 16:30:44.892328 23102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38241 every 8 connection(s)
I20260929 16:30:44.893102 23035 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-0/data/info.pb
I20260929 16:30:44.896093 23103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:44.900100 23103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:30:44.903250 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 23035
I20260929 16:30:44.903400 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-0/wal/instance
I20260929 16:30:44.903744 392 external_mini_cluster.cc:1428] Setting key df9f74931cc2c0f05756eaee338680f1
Sep 29 16:30:44 dist-test-slave-vq9v krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260929 16:30:44.916679 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:44.903501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34842 (local address 127.0.98.62:38241)
0929 16:30:44.903820 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:44.903831 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:44.903877 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:30:44.904488 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:44.904492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:44.904691 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:44.904924 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:44.905830 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:44.906692 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:44.907740 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:44.908125 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:44.908312 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:44.911120 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:44.911133 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:44.911148 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:44.911177 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:44.912974 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:44.913695 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:44.913701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:44.913703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:44.913828 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:44.914199 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:44.914204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:44.914209 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:44.914714 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:44.914870 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:44.915265 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 16:30:44.915536 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":98,"threads_started":1}
I20260929 16:30:44.916679 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:44.902500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38241 (local address 127.0.0.1:34842)
0929 16:30:44.903029 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:44.903077 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:44.904144 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:44.904918 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:44.904924 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:44.905413 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:44.905674 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:44.905690 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:44.906833 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:44.906836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:44.907607 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:44.907617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:44.908307 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:44.908311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:44.908400 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:44.909104 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:44.909121 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:44.910938 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:44.913102 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:44.913111 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:44.913120 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:44.913501 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:44.913934 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:44.913939 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:44.913941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:44.914059 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:44.914846 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:44.914854 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:44.915005 (+ 151us) client_negotiation.cc:769] Sending connection context
0929 16:30:44.915181 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 16:30:44.915586 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":118,"spinlock_wait_cycles":2944,"thread_start_us":127,"threads_started":1}
I20260929 16:30:44.919595 23103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:30:44.920433 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:44.920146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40295 (local address 127.0.0.1:36426)
0929 16:30:44.920275 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:44.920362 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 16:30:44.920760 23053 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40295: connect: Connection refused (error 111)
I20260929 16:30:44.921782 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:44.921558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40295 (local address 127.0.0.1:36428)
0929 16:30:44.921673 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:44.921726 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 16:30:44.922029 23103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40295: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40295: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 16:30:44.934194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40295
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38241,127.0.98.61:40295,127.0.98.60:38783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:30:44.971354 23103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } attempt: 1
I20260929 16:30:44.972348 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:44.971996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40295 (local address 127.0.0.1:36440)
0929 16:30:44.972180 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:44.972269 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 16:30:44.973126 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:44.972904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40295 (local address 127.0.0.1:36450)
0929 16:30:44.973017 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:44.973071 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 16:30:44.973287 23103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40295: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40295: connect: Connection refused (error 111)
I20260929 16:30:45.041448 23103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } attempt: 2
I20260929 16:30:45.042508 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.042198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40295 (local address 127.0.0.1:36456)
0929 16:30:45.042368 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:45.042432 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 16:30:45.043318 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.043085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40295 (local address 127.0.0.1:36470)
0929 16:30:45.043211 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:45.043263 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:30:45.043447 23103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40295: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40295: connect: Connection refused (error 111)
W20260929 16:30:45.050982 23110 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:30:45.051355 23110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:45.051419 23110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:45.051496 23110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:45.055857 23110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:45.055960 23110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:45.055996 23110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:45.056057 23110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:45.056113 23110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:45.061924 23110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38241,127.0.98.61:40295,127.0.98.60:38783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:45.063127 23110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:45.064280 23110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:45.073594 23118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:45.073607 23115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:45.073817 23110 server_base.cc:1034] running on GCE node
W20260929 16:30:45.073608 23116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:45.074661 23110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:45.075819 23110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:45.077011 23110 hybrid_clock.cc:648] HybridClock initialized: now 1790699445076972 us; error 50 us; skew 500 ppm
Sep 29 16:30:45 dist-test-slave-vq9v krb5kdc[23022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:45.080031 23110 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 16:30:45.081045 23110 webserver.cc:460] Webserver started at http://127.0.98.61:38667/ using document root <none> and password file <none>
I20260929 16:30:45.081719 23110 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:45.081789 23110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:45.081969 23110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:45.084098 23110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-1/data/instance:
uuid: "ecca13df1a5c453b8d7a307905538be8"
format_stamp: "Formatted at 2026-09-29 16:30:45 on dist-test-slave-vq9v"
server_key: "c54a55c5b5c0afc04c81370ba893d3aa"
server_key_iv: "e50089f052e69cf5c8cea4ed5df432fa"
server_key_version: "encryptionkey@0"
I20260929 16:30:45.084641 23110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-1/wal/instance:
uuid: "ecca13df1a5c453b8d7a307905538be8"
format_stamp: "Formatted at 2026-09-29 16:30:45 on dist-test-slave-vq9v"
server_key: "c54a55c5b5c0afc04c81370ba893d3aa"
server_key_iv: "e50089f052e69cf5c8cea4ed5df432fa"
server_key_version: "encryptionkey@0"
I20260929 16:30:45.088537 23110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:30:45.091245 23125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:45.092651 23110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:30:45.092828 23110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-1/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-1/wal
uuid: "ecca13df1a5c453b8d7a307905538be8"
format_stamp: "Formatted at 2026-09-29 16:30:45 on dist-test-slave-vq9v"
server_key: "c54a55c5b5c0afc04c81370ba893d3aa"
server_key_iv: "e50089f052e69cf5c8cea4ed5df432fa"
server_key_version: "encryptionkey@0"
I20260929 16:30:45.092962 23110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:45.118727 23110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:45.121933 23110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:45.122151 23110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:45.129449 23110 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40295
I20260929 16:30:45.129762 23177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40295 every 8 connection(s)
I20260929 16:30:45.130489 23110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-1/data/info.pb
I20260929 16:30:45.131592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 23110
I20260929 16:30:45.131690 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-1/wal/instance
I20260929 16:30:45.131994 392 external_mini_cluster.cc:1428] Setting key ef607fef9fea85ea66ab1d2182b9f980
I20260929 16:30:45.133514 23178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:45.134610 23103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } attempt: 3
I20260929 16:30:45.138023 23178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 29 16:30:45 dist-test-slave-vq9v krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 16:30:45 dist-test-slave-vq9v krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 16:30:45.152323 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.135168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40295 (local address 127.0.0.1:36486)
0929 16:30:45.135382 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:45.135399 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:45.135489 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:45.138292 (+ 2803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:45.138296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:45.138509 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:45.138652 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:45.138661 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.140743 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.140746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:45.141456 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:45.141465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.142523 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.142529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:45.142662 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:45.143247 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:45.143260 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:45.147081 (+ 3821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:45.150172 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:45.150178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:45.150182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:45.150488 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:45.150841 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:45.150844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:45.150846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:45.150967 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:45.151773 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:45.151775 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:45.151880 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 16:30:45.151993 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 16:30:45.152151 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260929 16:30:45.152586 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.139929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34858 (local address 127.0.98.62:38241)
0929 16:30:45.140071 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.140075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.140454 (+ 379us) server_negotiation.cc:407] Connection header received
0929 16:30:45.140660 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.140663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.140755 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.140832 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.141841 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.142558 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.143635 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.144098 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.144187 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:45.147925 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:45.147931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:45.147935 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:45.147966 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:45.150381 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.150958 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.150961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.150963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.151034 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.151378 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.151381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.151382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.151582 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:45.151686 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:45.152252 (+ 566us) server_negotiation.cc:299] Negotiation successful
0929 16:30:45.152402 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":69}
I20260929 16:30:45.153585 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.136185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36486 (local address 127.0.98.61:40295)
0929 16:30:45.136696 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.136716 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.136768 (+ 52us) server_negotiation.cc:407] Connection header received
0929 16:30:45.137572 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.137591 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.137960 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.138278 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.138830 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.140598 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.141580 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.142377 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.142528 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:45.147304 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:45.147314 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:45.147334 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:45.147370 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:45.150031 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.150582 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.150595 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.150598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.150715 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.151082 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.151088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.151093 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.151685 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:45.151810 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:45.152094 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 16:30:45.152437 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":332,"thread_start_us":136,"threads_started":1}
I20260929 16:30:45.153585 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.139865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38241 (local address 127.0.0.1:34858)
0929 16:30:45.140329 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:45.140381 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:45.140542 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:45.140918 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:45.140925 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:45.141476 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:45.141702 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:45.141710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.142707 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.142709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:45.143504 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:45.143514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.144267 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.144271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:45.144355 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:45.144892 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:45.144941 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:45.147746 (+ 2805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:45.150503 (+ 2757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:45.150515 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:45.150531 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:45.150855 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:45.151133 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:45.151139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:45.151142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:45.151263 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:45.151705 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:45.151715 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:45.151914 (+ 199us) client_negotiation.cc:769] Sending connection context
0929 16:30:45.152639 (+ 725us) client_negotiation.cc:240] Negotiation successful
0929 16:30:45.152861 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260929 16:30:45.155360 23178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:30:45.155886 23103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:30:45.156543 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.156327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38783 (local address 127.0.0.1:35546)
0929 16:30:45.156424 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:45.156481 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 16:30:45.157431 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.157065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38783 (local address 127.0.0.1:35558)
0929 16:30:45.157316 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:45.157366 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260929 16:30:45.157611 23103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38783: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38783: connect: Connection refused (error 111)
Sep 29 16:30:45 dist-test-slave-vq9v krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 16:30:45.164971 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.156070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40295 (local address 127.0.0.1:36490)
0929 16:30:45.156212 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:45.156251 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:45.156335 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:45.156878 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:45.156881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:45.157055 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:45.157172 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:45.157182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.158273 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.158276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:45.158826 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:45.158835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.159288 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.159291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:45.159356 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:45.159995 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:45.160007 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:45.161468 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:45.163304 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:45.163307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:45.163309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:45.163502 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:45.163851 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:45.163854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:45.163855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:45.163929 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:45.164447 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:45.164453 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:45.164576 (+ 123us) client_negotiation.cc:769] Sending connection context
0929 16:30:45.164713 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 16:30:45.164845 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 16:30:45.165144 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.156102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36490 (local address 127.0.98.61:40295)
0929 16:30:45.156261 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.156265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.156307 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:30:45.156449 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.156455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.156501 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.156633 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.157349 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.158124 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.158962 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.159185 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.159266 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:45.161679 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:45.161682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:45.161684 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:45.161706 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:45.163185 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.163670 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.163674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.163675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.163758 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.164042 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.164045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.164047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.164313 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:45.164465 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:45.164848 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 16:30:45.164989 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:30:45.165563 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0hb3v3/build/debug/bin/kudu
/tmp/dist-test-task0hb3v3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38783
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38241,127.0.98.61:40295,127.0.98.60:38783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:30:45.165956 23178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:30:45.166697 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.166440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38783 (local address 127.0.0.1:35566)
0929 16:30:45.166563 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:45.166642 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:30:45.167413 23127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38783: connect: Connection refused (error 111)
I20260929 16:30:45.168426 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.168095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38783 (local address 127.0.0.1:35578)
0929 16:30:45.168271 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:45.168343 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260929 16:30:45.168578 23178 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38783: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38783: connect: Connection refused (error 111)
I20260929 16:30:45.188823 23103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } attempt: 1
I20260929 16:30:45.189810 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.189469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38783 (local address 127.0.0.1:35582)
0929 16:30:45.189626 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:45.189696 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 16:30:45.190593 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.190372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38783 (local address 127.0.0.1:35592)
0929 16:30:45.190483 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:45.190535 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 16:30:45.190752 23103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38783: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38783: connect: Connection refused (error 111)
I20260929 16:30:45.217862 23178 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } attempt: 1
I20260929 16:30:45.218861 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.218496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38783 (local address 127.0.0.1:35600)
0929 16:30:45.218693 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:45.218767 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 16:30:45.219679 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.219411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38783 (local address 127.0.0.1:35606)
0929 16:30:45.219532 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:45.219590 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 16:30:45.219844 23178 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38783: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38783: connect: Connection refused (error 111)
I20260929 16:30:45.265919 23103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } attempt: 2
I20260929 16:30:45.266826 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.266498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38783 (local address 127.0.0.1:35614)
0929 16:30:45.266645 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:45.266723 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 16:30:45.267578 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.267353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38783 (local address 127.0.0.1:35620)
0929 16:30:45.267465 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:45.267528 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:30:45.267704 23103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38783: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38783: connect: Connection refused (error 111)
W20260929 16:30:45.282282 23184 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:30:45.282637 23184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:30:45.282709 23184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:30:45.282795 23184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:30:45.286972 23184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:30:45.287065 23184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:30:45.287103 23184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:30:45.287141 23184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:30:45.287180 23184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:30:45.288062 23178 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } attempt: 2
I20260929 16:30:45.288965 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.288630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38783 (local address 127.0.0.1:35632)
0929 16:30:45.288804 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:45.288874 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 16:30:45.289778 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.289548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38783 (local address 127.0.0.1:35648)
0929 16:30:45.289657 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:45.289708 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 16:30:45.289927 23178 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38783: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38783: connect: Connection refused (error 111)
I20260929 16:30:45.293480 23184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38241,127.0.98.61:40295,127.0.98.60:38783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 15:02:05 UTC on ccycloud.quasar-dtftcp.root.comops.site
build id 4679901
I20260929 16:30:45.294610 23184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:30:45.295646 23184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:30:45.304493 23191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:45.304543 23190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:30:45.304538 23193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:30:45.304870 23184 server_base.cc:1034] running on GCE node
I20260929 16:30:45.305464 23184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:30:45.306622 23184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:30:45.307793 23184 hybrid_clock.cc:648] HybridClock initialized: now 1790699445307778 us; error 31 us; skew 500 ppm
Sep 29 16:30:45 dist-test-slave-vq9v krb5kdc[23022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:30:45.310813 23184 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 16:30:45.311686 23184 webserver.cc:460] Webserver started at http://127.0.98.60:41957/ using document root <none> and password file <none>
I20260929 16:30:45.312228 23184 fs_manager.cc:359] Metadata directory not provided
I20260929 16:30:45.312284 23184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:30:45.312448 23184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:30:45.314581 23184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-2/data/instance:
uuid: "5e190be720a6474cb544a49953a72394"
format_stamp: "Formatted at 2026-09-29 16:30:45 on dist-test-slave-vq9v"
server_key: "ba769045a6e276f91c75c6c5f2faba3f"
server_key_iv: "a25fae18cec608b8b10299219c938c76"
server_key_version: "encryptionkey@0"
I20260929 16:30:45.315088 23184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-2/wal/instance:
uuid: "5e190be720a6474cb544a49953a72394"
format_stamp: "Formatted at 2026-09-29 16:30:45 on dist-test-slave-vq9v"
server_key: "ba769045a6e276f91c75c6c5f2faba3f"
server_key_iv: "a25fae18cec608b8b10299219c938c76"
server_key_version: "encryptionkey@0"
I20260929 16:30:45.318791 23184 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:30:45.321278 23200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:30:45.322546 23184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:30:45.322690 23184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-2/data,/tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-2/wal
uuid: "5e190be720a6474cb544a49953a72394"
format_stamp: "Formatted at 2026-09-29 16:30:45 on dist-test-slave-vq9v"
server_key: "ba769045a6e276f91c75c6c5f2faba3f"
server_key_iv: "a25fae18cec608b8b10299219c938c76"
server_key_version: "encryptionkey@0"
I20260929 16:30:45.322808 23184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:30:45.343822 23184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:30:45.347322 23184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:30:45.347541 23184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:30:45.354652 23184 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38783
I20260929 16:30:45.354962 23252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38783 every 8 connection(s)
I20260929 16:30:45.355684 23184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-2/data/info.pb
I20260929 16:30:45.358587 23253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:30:45.362372 23253 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:30:45.362988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0hb3v3/build/debug/bin/kudu as pid 23184
I20260929 16:30:45.363111 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0hb3v3/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790699152396020-392-0/minicluster-data/master-2/wal/instance
I20260929 16:30:45.363425 392 external_mini_cluster.cc:1428] Setting key 905cba6f8cc85cd3365fecefd8d09015
I20260929 16:30:45.366865 23103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } attempt: 3
Sep 29 16:30:45 dist-test-slave-vq9v krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 16:30:45 dist-test-slave-vq9v krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 16:30:45 dist-test-slave-vq9v krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 16:30:45.376982 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.364674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34874 (local address 127.0.98.62:38241)
0929 16:30:45.364809 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.364814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.365476 (+ 662us) server_negotiation.cc:407] Connection header received
0929 16:30:45.366286 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.366290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.366355 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.366456 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.367524 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.368085 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.369517 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.369882 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.369982 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:45.372744 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:45.372748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:45.372752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:45.372782 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:45.374638 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.375134 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.375139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.375141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.375228 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.375625 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.375631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.375635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.375882 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:45.375994 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:45.376644 (+ 650us) server_negotiation.cc:299] Negotiation successful
0929 16:30:45.376775 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:30:45.378350 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.364747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35654 (local address 127.0.98.60:38783)
0929 16:30:45.365171 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.365193 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.365264 (+ 71us) server_negotiation.cc:407] Connection header received
0929 16:30:45.366065 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.366083 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.366299 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.366476 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.367390 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.368344 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.369445 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.369902 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.370068 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:45.373431 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:45.373439 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:45.373454 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:45.373483 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:45.375608 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.376024 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.376029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.376031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.376112 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.376392 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.376397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.376399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.376633 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:45.376850 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:45.377049 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 16:30:45.377402 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":290,"spinlock_wait_cycles":896,"thread_start_us":108,"threads_started":1}
I20260929 16:30:45.378366 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.364747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38241 (local address 127.0.0.1:34874)
0929 16:30:45.365338 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:45.365377 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:45.366078 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:45.366528 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:45.366533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:45.366976 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:45.367346 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:45.367364 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.368202 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.368204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:45.369340 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:45.369355 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.370036 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.370040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:45.370193 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:45.370858 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:45.370882 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:45.372602 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:45.374748 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:45.374758 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:45.374769 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:45.375029 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:45.375325 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:45.375328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:45.375330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:45.375519 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:45.375993 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:45.376000 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:45.376341 (+ 341us) client_negotiation.cc:769] Sending connection context
0929 16:30:45.377193 (+ 852us) client_negotiation.cc:240] Negotiation successful
0929 16:30:45.377467 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":89,"spinlock_wait_cycles":37376,"thread_start_us":177,"threads_started":1}
I20260929 16:30:45.378647 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.367299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38783 (local address 127.0.0.1:35660)
0929 16:30:45.367484 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:45.367498 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:45.367587 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:45.368837 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:45.368840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:45.369041 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:45.369133 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:45.369140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.370118 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.370124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:45.370771 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:45.370781 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.371429 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.371435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:45.371535 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:45.372020 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:45.372035 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:45.373710 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:45.376802 (+ 3092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:45.376805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:45.376807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:45.377023 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:45.377633 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:45.377636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:45.377638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:45.377729 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:45.378155 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:45.378158 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:45.378255 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 16:30:45.378373 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 16:30:45.378508 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260929 16:30:45.379756 23253 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:30:45.380316 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.367361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35660 (local address 127.0.98.60:38783)
0929 16:30:45.368598 (+ 1237us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.368607 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.368619 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:30:45.368648 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.368651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.368691 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.368814 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.369281 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.370033 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.370918 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.371284 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.371404 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:45.373877 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:45.373882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:45.373889 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:45.373910 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:45.376695 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.377147 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.377150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.377152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.377527 (+ 375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.377846 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.377849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.377851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.378037 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:45.378135 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:45.378448 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 16:30:45.380122 (+ 1674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1163,"thread_start_us":116,"threads_started":1}
I20260929 16:30:45.381151 23178 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } attempt: 3
I20260929 16:30:45.383644 23103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d: Bootstrap starting.
Sep 29 16:30:45 dist-test-slave-vq9v krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 16:30:45 dist-test-slave-vq9v krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 16:30:45.386662 23103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:45.387624 23103 log.cc:824] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:30:45 dist-test-slave-vq9v krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 16:30:45.390058 23103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d: No bootstrap required, opened a new log
I20260929 16:30:45.391304 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.380584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.0.98.61:40295)
0929 16:30:45.380728 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.380733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.380746 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:30:45.380791 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.380795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.380846 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.380927 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.381927 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.382473 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.383210 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.383445 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.383555 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:45.387186 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:45.387190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:45.387193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:45.387221 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:45.389115 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.389776 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.389781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.389784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.389872 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.390289 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.390296 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.390299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.390541 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:45.390632 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:45.391004 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 16:30:45.391141 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:30:45.392063 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.380254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40295 (local address 127.0.0.1:36506)
0929 16:30:45.380534 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:45.380549 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:45.380626 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:45.381491 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:45.381494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:45.381667 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:45.381796 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:45.381807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.382586 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.382589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:45.383090 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:45.383095 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.383556 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.383560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:45.383658 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:45.384448 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:45.384464 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:45.387024 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:45.389369 (+ 2345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:45.389372 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:45.389375 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:45.389639 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:45.389991 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:45.389996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:45.389999 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:45.390093 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:45.390692 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:45.390694 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:45.390811 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 16:30:45.391745 (+ 934us) client_negotiation.cc:240] Negotiation successful
0929 16:30:45.391900 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"spinlock_wait_cycles":896}
I20260929 16:30:45.392143 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.380912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34878 (local address 127.0.98.62:38241)
0929 16:30:45.381019 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.381023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.381033 (+ 10us) server_negotiation.cc:407] Connection header received
0929 16:30:45.381065 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.381068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.381107 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.381183 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.381996 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.382699 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.383749 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.384038 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.384138 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:45.387186 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:45.387190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:45.387193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:45.387232 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:45.389663 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.390693 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.390696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.390698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.390783 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.391106 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.391111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.391113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.391337 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:45.391512 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:45.391886 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 16:30:45.391997 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 16:30:45.392891 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.381554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38783 (local address 127.0.0.1:35672)
0929 16:30:45.381773 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:45.381788 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:45.381861 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:45.384511 (+ 2650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:45.384514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:45.384693 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:45.384784 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:45.384791 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.385977 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.385980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:45.386745 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:45.386757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.387397 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.387403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:45.387503 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:45.387996 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:45.388017 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:45.389429 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:45.391352 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:45.391355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:45.391357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:45.391613 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:45.391918 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:45.391923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:45.391925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:45.392005 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:45.392427 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:45.392431 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:45.392528 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 16:30:45.392608 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 16:30:45.392737 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260929 16:30:45.393478 23253 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:30:45.393580 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.381985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35672 (local address 127.0.98.60:38783)
0929 16:30:45.384285 (+ 2300us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.384301 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.384314 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:30:45.384346 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.384349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.384386 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.384896 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.384995 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.385743 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.386900 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.387245 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.387700 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:45.389579 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:45.389583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:45.389586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:45.389612 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:45.391203 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.391731 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.391733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.391735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.391814 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.392109 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.392111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.392113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.392312 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:45.393159 (+ 847us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:45.393264 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 16:30:45.393392 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2228}
I20260929 16:30:45.394899 23103 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } } peers { permanent_uuid: "ecca13df1a5c453b8d7a307905538be8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } } peers { permanent_uuid: "5e190be720a6474cb544a49953a72394" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } }
I20260929 16:30:45.395177 23103 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5eb2092a8fb43e8aa80a13c9f2a910d, State: Initialized, Role: FOLLOWER
I20260929 16:30:45.395545 23103 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } } peers { permanent_uuid: "ecca13df1a5c453b8d7a307905538be8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } } peers { permanent_uuid: "5e190be720a6474cb544a49953a72394" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } }
I20260929 16:30:45.396086 23260 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } } peers { permanent_uuid: "ecca13df1a5c453b8d7a307905538be8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } } peers { permanent_uuid: "5e190be720a6474cb544a49953a72394" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } } }
I20260929 16:30:45.396314 23260 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:30:45.397063 23103 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:45.397396 23178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecca13df1a5c453b8d7a307905538be8: Bootstrap starting.
Sep 29 16:30:45 dist-test-slave-vq9v krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 16:30:45.401472 23178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecca13df1a5c453b8d7a307905538be8: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:45.402422 23178 log.cc:824] T 00000000000000000000000000000000 P ecca13df1a5c453b8d7a307905538be8: Log is configured to *not* fsync() on all Append() calls
W20260929 16:30:45.402587 23271 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:30:45.402655 23271 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:30:45.404529 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.393708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36518 (local address 127.0.98.61:40295)
0929 16:30:45.393817 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.393823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.393933 (+ 110us) server_negotiation.cc:407] Connection header received
0929 16:30:45.394116 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.394120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.394167 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.394255 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.395056 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.395644 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.396803 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.397035 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.397202 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:45.400164 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:45.400168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:45.400171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:45.400200 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:45.401961 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.402665 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.402669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.402670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.402761 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.403098 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.403102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.403103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.403350 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:45.403539 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:45.404265 (+ 726us) server_negotiation.cc:299] Negotiation successful
0929 16:30:45.404377 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:30:45.404768 23178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecca13df1a5c453b8d7a307905538be8: No bootstrap required, opened a new log
I20260929 16:30:45.407743 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 29 16:30:45 dist-test-slave-vq9v krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790699445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 16:30:45.409035 23178 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecca13df1a5c453b8d7a307905538be8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } } peers { permanent_uuid: "ecca13df1a5c453b8d7a307905538be8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } } peers { permanent_uuid: "5e190be720a6474cb544a49953a72394" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } }
I20260929 16:30:45.409345 23178 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecca13df1a5c453b8d7a307905538be8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecca13df1a5c453b8d7a307905538be8, State: Initialized, Role: FOLLOWER
I20260929 16:30:45.409737 23178 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecca13df1a5c453b8d7a307905538be8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } } peers { permanent_uuid: "ecca13df1a5c453b8d7a307905538be8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } } peers { permanent_uuid: "5e190be720a6474cb544a49953a72394" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } }
I20260929 16:30:45.409725 23077 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34878
I20260929 16:30:45.409963 23077 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 16:30:45.410079 23077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:45.411170 23178 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecca13df1a5c453b8d7a307905538be8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:45.411422 23077 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } } peers { permanent_uuid: "ecca13df1a5c453b8d7a307905538be8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } } peers { permanent_uuid: "5e190be720a6474cb544a49953a72394" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } }
I20260929 16:30:45.411902 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.394006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38783 (local address 127.0.0.1:35688)
0929 16:30:45.394749 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0929 16:30:45.394763 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:30:45.394837 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:30:45.403234 (+ 8397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:30:45.403238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:30:45.403446 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:30:45.403565 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:45.403575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.404557 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.404562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:45.405329 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:30:45.405340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.406222 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.406228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:30:45.406326 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:45.406838 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:30:45.406852 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:30:45.408399 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:30:45.410347 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:45.410350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:45.410352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:45.410590 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:45.410884 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:30:45.410886 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:30:45.410888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:30:45.410966 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:30:45.411394 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:30:45.411396 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:30:45.411476 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 16:30:45.411572 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 16:30:45.411717 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659}
I20260929 16:30:45.412473 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.394621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35688 (local address 127.0.98.60:38783)
0929 16:30:45.402982 (+ 8361us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.402986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.403001 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:30:45.403048 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.403053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.403112 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.403667 (+ 555us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.403737 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.404448 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.405457 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.406103 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.406547 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:45.408535 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:45.408538 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:45.408540 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:45.408563 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:45.410231 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.410694 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.410697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.410699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.410777 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.411055 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.411058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.411060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.411268 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:45.411360 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:45.412207 (+ 847us) server_negotiation.cc:299] Negotiation successful
0929 16:30:45.412328 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8254}
I20260929 16:30:45.412499 23077 leader_election.cc:288] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d [CANDIDATE]: Term 1 election: Requested vote from peers ecca13df1a5c453b8d7a307905538be8 (127.0.98.61:40295), 5e190be720a6474cb544a49953a72394 (127.0.98.60:38783)
I20260929 16:30:45.413954 23152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ecca13df1a5c453b8d7a307905538be8"
I20260929 16:30:45.414208 23152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecca13df1a5c453b8d7a307905538be8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:45.402391 23035 master_runner.cc:186] Error getting master registration for 127.0.98.61:40295: OK, instance_id { permanent_uuid: "ecca13df1a5c453b8d7a307905538be8" instance_seqno: 1790699445127439 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:30:45.414915 23035 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:30:45.415627 23152 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ecca13df1a5c453b8d7a307905538be8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5eb2092a8fb43e8aa80a13c9f2a910d in term 1.
I20260929 16:30:45.415863 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecca13df1a5c453b8d7a307905538be8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } } peers { permanent_uuid: "ecca13df1a5c453b8d7a307905538be8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } } peers { permanent_uuid: "5e190be720a6474cb544a49953a72394" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } } }
I20260929 16:30:45.416070 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecca13df1a5c453b8d7a307905538be8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:30:45.416105 23053 leader_election.cc:302] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5eb2092a8fb43e8aa80a13c9f2a910d, ecca13df1a5c453b8d7a307905538be8; no voters:
I20260929 16:30:45.416425 23260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:30:45.416667 23260 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d [term 1 LEADER]: Becoming Leader. State: Replica: d5eb2092a8fb43e8aa80a13c9f2a910d, State: Running, Role: LEADER
I20260929 16:30:45.417007 23260 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } } peers { permanent_uuid: "ecca13df1a5c453b8d7a307905538be8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } } peers { permanent_uuid: "5e190be720a6474cb544a49953a72394" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } }
I20260929 16:30:45.418489 23253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e190be720a6474cb544a49953a72394: Bootstrap starting.
I20260929 16:30:45.419670 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5eb2092a8fb43e8aa80a13c9f2a910d. Latest consensus state: current_term: 1 leader_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } } peers { permanent_uuid: "ecca13df1a5c453b8d7a307905538be8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } } peers { permanent_uuid: "5e190be720a6474cb544a49953a72394" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } } }
I20260929 16:30:45.419986 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:45.420472 23291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:30:45.421552 23253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e190be720a6474cb544a49953a72394: Neither blocks nor log segments found. Creating new log.
I20260929 16:30:45.421586 23291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:30:45.422485 23253 log.cc:824] T 00000000000000000000000000000000 P 5e190be720a6474cb544a49953a72394: Log is configured to *not* fsync() on all Append() calls
I20260929 16:30:45.425087 23253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e190be720a6474cb544a49953a72394: No bootstrap required, opened a new log
W20260929 16:30:45.432178 23293 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ecca13df1a5c453b8d7a307905538be8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:30:45.432286 23293 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:30:45.432775 23253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e190be720a6474cb544a49953a72394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } } peers { permanent_uuid: "ecca13df1a5c453b8d7a307905538be8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } } peers { permanent_uuid: "5e190be720a6474cb544a49953a72394" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } }
I20260929 16:30:45.433058 23253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e190be720a6474cb544a49953a72394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e190be720a6474cb544a49953a72394, State: Initialized, Role: FOLLOWER
I20260929 16:30:45.428911 23110 master_runner.cc:186] Error getting master registration for 127.0.98.60:38783: OK, instance_id { permanent_uuid: "5e190be720a6474cb544a49953a72394" instance_seqno: 1790699445352677 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:30:45.433529 23110 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:30:45.433454 23253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e190be720a6474cb544a49953a72394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } } peers { permanent_uuid: "ecca13df1a5c453b8d7a307905538be8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } } peers { permanent_uuid: "5e190be720a6474cb544a49953a72394" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } }
I20260929 16:30:45.434192 23296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e190be720a6474cb544a49953a72394 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } } peers { permanent_uuid: "ecca13df1a5c453b8d7a307905538be8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } } peers { permanent_uuid: "5e190be720a6474cb544a49953a72394" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } } }
I20260929 16:30:45.434494 23296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e190be720a6474cb544a49953a72394 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:30:45.435072 23253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e190be720a6474cb544a49953a72394 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:30:45.435138 23227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5e190be720a6474cb544a49953a72394"
I20260929 16:30:45.435364 23227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e190be720a6474cb544a49953a72394 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:30:45.436947 23227 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e190be720a6474cb544a49953a72394 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5eb2092a8fb43e8aa80a13c9f2a910d in term 1.
I20260929 16:30:45.447962 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.437791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35696 (local address 127.0.98.60:38783)
0929 16:30:45.437915 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.437918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.438432 (+ 514us) server_negotiation.cc:407] Connection header received
0929 16:30:45.438649 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.438652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.438705 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.438761 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.439844 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.440616 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.441828 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.442150 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.442232 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:45.444137 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:45.444144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:45.444147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:45.444175 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:45.446118 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.446599 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.446602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.446604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.446684 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.446986 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.446990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.446992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.447196 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:45.447329 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:45.447651 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 16:30:45.447779 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:30:45.454005 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.439301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.0.98.61:40295)
0929 16:30:45.439453 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.439458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.444670 (+ 5212us) server_negotiation.cc:407] Connection header received
0929 16:30:45.444891 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.444895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.444960 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.445025 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.445964 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.446697 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.447768 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.448076 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.448164 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:45.450141 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:45.450147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:45.450150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:45.450182 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:45.451883 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.452432 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.452435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.452437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.452525 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.452851 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.452855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.452856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.453094 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:45.453260 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:45.453678 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 16:30:45.453801 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 16:30:45.456835 23310 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e190be720a6474cb544a49953a72394: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:30:45.456947 23310 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:30:45.458390 23152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ecca13df1a5c453b8d7a307905538be8 [term 1 FOLLOWER]: Refusing update from remote peer d5eb2092a8fb43e8aa80a13c9f2a910d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:45.458390 23227 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e190be720a6474cb544a49953a72394 [term 1 FOLLOWER]: Refusing update from remote peer d5eb2092a8fb43e8aa80a13c9f2a910d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:30:45.459057 23260 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecca13df1a5c453b8d7a307905538be8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:45.459398 23289 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e190be720a6474cb544a49953a72394" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:30:45.462823 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:45.438595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34888 (local address 127.0.98.62:38241)
0929 16:30:45.438714 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.438731 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.440228 (+ 1497us) server_negotiation.cc:407] Connection header received
0929 16:30:45.440394 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.440397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.440447 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.440516 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.441673 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.442403 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.443546 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:45.443945 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:45.444037 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:45.457231 (+ 13194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:30:45.457236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:30:45.457239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:30:45.457268 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:30:45.460180 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.461351 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.461355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.461357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.461446 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:30:45.461772 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:30:45.461776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:30:45.461777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:30:45.462000 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:30:45.462108 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:45.462530 (+ 422us) server_negotiation.cc:299] Negotiation successful
0929 16:30:45.462656 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:30:45.464120 23296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e190be720a6474cb544a49953a72394 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5eb2092a8fb43e8aa80a13c9f2a910d. Latest consensus state: current_term: 1 leader_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } } peers { permanent_uuid: "ecca13df1a5c453b8d7a307905538be8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } } peers { permanent_uuid: "5e190be720a6474cb544a49953a72394" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } } }
I20260929 16:30:45.464273 23296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e190be720a6474cb544a49953a72394 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:30:45.468838 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecca13df1a5c453b8d7a307905538be8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5eb2092a8fb43e8aa80a13c9f2a910d. Latest consensus state: current_term: 1 leader_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } } peers { permanent_uuid: "ecca13df1a5c453b8d7a307905538be8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } } peers { permanent_uuid: "5e190be720a6474cb544a49953a72394" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } } }
I20260929 16:30:45.468977 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecca13df1a5c453b8d7a307905538be8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:30:45.478662 23260 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } } peers { permanent_uuid: "ecca13df1a5c453b8d7a307905538be8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } } peers { permanent_uuid: "5e190be720a6474cb544a49953a72394" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } } }
I20260929 16:30:45.478811 23260 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:45.480161 23296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e190be720a6474cb544a49953a72394 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } } peers { permanent_uuid: "ecca13df1a5c453b8d7a307905538be8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } } peers { permanent_uuid: "5e190be720a6474cb544a49953a72394" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } } }
I20260929 16:30:45.480291 23296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e190be720a6474cb544a49953a72394 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:30:45.482474 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } } peers { permanent_uuid: "ecca13df1a5c453b8d7a307905538be8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } } peers { permanent_uuid: "5e190be720a6474cb544a49953a72394" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } } }
I20260929 16:30:45.482632 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d [sys.catalog]: This master's current role is: LEADER
I20260929 16:30:45.483693 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecca13df1a5c453b8d7a307905538be8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5eb2092a8fb43e8aa80a13c9f2a910d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38241 } } peers { permanent_uuid: "ecca13df1a5c453b8d7a307905538be8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40295 } } peers { permanent_uuid: "5e190be720a6474cb544a49953a72394" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38783 } } }
I20260929 16:30:45.483822 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecca13df1a5c453b8d7a307905538be8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:30:45.487018 23314 mvcc.cc:205] Tried to move back new op lower bound from 7334704928571559936 to 7334704928432205824. Current Snapshot: MvccSnapshot[applied={T|T < 7334704928571559936}]
I20260929 16:30:45.487200 23312 mvcc.cc:205] Tried to move back new op lower bound from 7334704928571559936 to 7334704928432205824. Current Snapshot: MvccSnapshot[applied={T|T < 7334704928571559936}]
I20260929 16:30:45.488512 23291 catalog_manager.cc:1347] Generated new cluster ID: a114f46626e641dd9ee9869ead79c8cc
I20260929 16:30:45.488579 23291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:30:45.535220 23291 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:30:45.536789 23291 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:30:45.563875 23291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5eb2092a8fb43e8aa80a13c9f2a910d: Generated new TSK 0
I20260929 16:30:45.564860 23291 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:30:45.601517 23106 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:45.600047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34890 (local address 127.0.98.62:38241)
0929 16:30:45.600306 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.600312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.600382 (+ 70us) server_negotiation.cc:407] Connection header received
0929 16:30:45.600599 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.600603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.600658 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.600710 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.601379 (+ 669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34890: BlockingRecv error: recv got EOF from 127.0.0.1:34890 (error 108)
Metrics: {"server-negotiator.queue_time_us":172}
W20260929 16:30:45.602985 23181 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:45.600168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36538 (local address 127.0.98.61:40295)
0929 16:30:45.600407 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.600411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.602124 (+ 1713us) server_negotiation.cc:407] Connection header received
0929 16:30:45.602288 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.602291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.602338 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.602387 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.602864 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36538: BlockingRecv error: recv got EOF from 127.0.0.1:36538 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
W20260929 16:30:45.604367 23258 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:45.598957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35712 (local address 127.0.98.60:38783)
0929 16:30:45.599086 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.599091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.603543 (+ 4452us) server_negotiation.cc:407] Connection header received
0929 16:30:45.603702 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.603705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.603751 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.603811 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.604254 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35712: BlockingRecv error: recv got EOF from 127.0.0.1:35712 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 16:30:45.609319 23181 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:45.607643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36542 (local address 127.0.98.61:40295)
0929 16:30:45.607747 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.607751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.608265 (+ 514us) server_negotiation.cc:407] Connection header received
0929 16:30:45.608488 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.608491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.608566 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.608621 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.609148 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36542: BlockingRecv error: recv got EOF from 127.0.0.1:36542 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 16:30:45.614720 23181 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:45.613101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36556 (local address 127.0.98.61:40295)
0929 16:30:45.613239 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.613243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.613758 (+ 515us) server_negotiation.cc:407] Connection header received
0929 16:30:45.613964 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.613969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.614021 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.614073 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.614612 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36556: BlockingRecv error: recv got EOF from 127.0.0.1:36556 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 16:30:45.620712 23181 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:45.619117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36564 (local address 127.0.98.61:40295)
0929 16:30:45.619225 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.619230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.619764 (+ 534us) server_negotiation.cc:407] Connection header received
0929 16:30:45.619978 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.619983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.620037 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.620090 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.620603 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36564: BlockingRecv error: recv got EOF from 127.0.0.1:36564 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 16:30:45.629007 23181 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:45.627226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36574 (local address 127.0.98.61:40295)
0929 16:30:45.627371 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.627376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.627933 (+ 557us) server_negotiation.cc:407] Connection header received
0929 16:30:45.628162 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.628166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.628235 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.628296 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.628886 (+ 590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36574: BlockingRecv error: recv got EOF from 127.0.0.1:36574 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 16:30:45.641523 23181 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:45.639692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36582 (local address 127.0.98.61:40295)
0929 16:30:45.639843 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.639848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.640482 (+ 634us) server_negotiation.cc:407] Connection header received
0929 16:30:45.640694 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.640698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.640759 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.640810 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.641405 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36582: BlockingRecv error: recv got EOF from 127.0.0.1:36582 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 16:30:45.661330 23181 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:45.659940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36586 (local address 127.0.98.61:40295)
0929 16:30:45.660081 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.660085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.660274 (+ 189us) server_negotiation.cc:407] Connection header received
0929 16:30:45.660464 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.660472 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.660527 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.660649 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.661153 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36586: BlockingRecv error: recv got EOF from 127.0.0.1:36586 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 16:30:45.696720 23181 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:45.695494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36588 (local address 127.0.98.61:40295)
0929 16:30:45.695644 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.695651 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.695805 (+ 154us) server_negotiation.cc:407] Connection header received
0929 16:30:45.695931 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.695934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.695984 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.696058 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.696576 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36588: BlockingRecv error: recv got EOF from 127.0.0.1:36588 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 16:30:45.764547 23181 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:45.763345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36600 (local address 127.0.98.61:40295)
0929 16:30:45.763518 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.763522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.763647 (+ 125us) server_negotiation.cc:407] Connection header received
0929 16:30:45.763830 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.763833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.763885 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.763971 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.764404 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36600: BlockingRecv error: recv got EOF from 127.0.0.1:36600 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260929 16:30:45.896240 23181 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:45.894955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36606 (local address 127.0.98.61:40295)
0929 16:30:45.895084 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:45.895091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:45.895290 (+ 199us) server_negotiation.cc:407] Connection header received
0929 16:30:45.895471 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:45.895473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:45.895523 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:45.895622 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:45.896130 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36606: BlockingRecv error: recv got EOF from 127.0.0.1:36606 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 16:30:46.156144 23181 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:30:46.155054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40836 (local address 127.0.98.61:40295)
0929 16:30:46.155213 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:46.155218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:46.155236 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:30:46.155373 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:46.155376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:46.155423 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:46.155518 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:30:46.155996 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40836: BlockingRecv error: recv got EOF from 127.0.0.1:40836 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:30:46.433786 23293 catalog_manager.cc:1259] Loaded cluster ID: a114f46626e641dd9ee9869ead79c8cc
I20260929 16:30:46.433876 23293 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ecca13df1a5c453b8d7a307905538be8: loading cluster ID for follower catalog manager: success
I20260929 16:30:46.435771 23293 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ecca13df1a5c453b8d7a307905538be8: acquiring CA information for follower catalog manager: success
I20260929 16:30:46.437296 23293 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ecca13df1a5c453b8d7a307905538be8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:30:46.458376 23310 catalog_manager.cc:1259] Loaded cluster ID: a114f46626e641dd9ee9869ead79c8cc
I20260929 16:30:46.458465 23310 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5e190be720a6474cb544a49953a72394: loading cluster ID for follower catalog manager: success
I20260929 16:30:46.460135 23310 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5e190be720a6474cb544a49953a72394: acquiring CA information for follower catalog manager: success
I20260929 16:30:46.461426 23310 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5e190be720a6474cb544a49953a72394: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:30:46.674433 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:46.670544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.0.98.61:40295)
0929 16:30:46.670801 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:46.670831 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:46.670847 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:30:46.670902 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:46.670905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:46.671021 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:46.671126 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:30:46.671374 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:46.671957 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:46.672811 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:46.673039 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:46.673122 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:46.673501 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:30:46.673979 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:30:46.674084 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:46.674135 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 16:30:46.674186 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
I20260929 16:30:46.682308 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:46.677499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.0.98.60:38783)
0929 16:30:46.677767 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:46.677775 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:46.677834 (+ 59us) server_negotiation.cc:407] Connection header received
0929 16:30:46.678022 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:46.678025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:46.678154 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:46.678307 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:30:46.678527 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:46.679064 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:46.680399 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:46.680803 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:46.680932 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:46.681320 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:30:46.681814 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:30:46.681946 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:46.682038 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 16:30:46.682121 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":75,"threads_started":1}
I20260929 16:30:46.690225 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:46.685665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.0.98.60:38783)
0929 16:30:46.685787 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:46.685792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:46.686118 (+ 326us) server_negotiation.cc:407] Connection header received
0929 16:30:46.686248 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:46.686251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:46.686304 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:46.686383 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:30:46.686646 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:46.687455 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:46.688856 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:46.689154 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:46.689305 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:46.689524 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:30:46.689602 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:30:46.689676 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:46.690000 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 16:30:46.690066 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:30:46.690333 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:46.685677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57432 (local address 127.0.98.62:38241)
0929 16:30:46.686182 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:46.686186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:46.686201 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:30:46.686285 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:46.686288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:46.686466 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:46.686592 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:30:46.686913 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:46.687473 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:46.688519 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:46.688762 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:46.688883 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:46.689366 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:30:46.689740 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:30:46.689823 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:46.690084 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 16:30:46.690149 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":80,"threads_started":1}
I20260929 16:30:46.690591 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:30:46.685901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40856 (local address 127.0.98.61:40295)
0929 16:30:46.686031 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 16:30:46.686035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:30:46.686433 (+ 398us) server_negotiation.cc:407] Connection header received
0929 16:30:46.686603 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:30:46.686606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:30:46.686663 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:30:46.686743 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:30:46.686981 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:46.687694 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:46.689377 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:30:46.689678 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:30:46.689769 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:30:46.690075 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:30:46.690154 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:30:46.690200 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 16:30:46.690387 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 16:30:46.690451 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:30:46.693881 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 23035
I20260929 16:30:46.700829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 23110
I20260929 16:30:46.707118 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0hb3v3/build/debug/bin/kudu with pid 23184
2026-09-29T16:30:46Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3174 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6358 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (294302 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST